String Accessor

Core Principle: The .str accessor provides vectorized string manipulation routines on Series objects, facilitating regular expression extraction, string splitting, and delimiter expansion over packed biological fields.

The basics

s.str.upper(); s.str.lower(); s.str.strip()
s.str.len()
s.str.startswith("chr"); s.str.endswith("_1")
s.str.contains("BRCA")                          # regex by default
s.str.contains("BRCA", regex=False)             # literal, faster
s.str.replace("chr", "", regex=False)
s.str.removeprefix("chr"); s.str.removesuffix(".bam")
s.str.split(":")                                # → Series of lists
s.str.split(":", expand=True)                   # → DataFrame of columns
s.str.split(":", n=1, expand=True)              # split only on the first occurrence
s.str.cat(sep=",")                              # join all values
s.str[0:3]                                      # slice each string
s.str.zfill(2)                                  # '1' → '01'
s.str.pad(10, side="left")

.str only works on string dtype. Calling it on a numeric column raises, which is a free type check.

Note that in pandas 3.0 string columns are str dtype (Arrow-backed) rather than object. .str works the same way but is considerably faster. See pandas dtypes.

Extraction with regex

s.str.extract(r"(\w+):(\d+)")                   # first match → DataFrame of groups
s.str.extract(r"(?P<chrom>\w+):(?P<pos>\d+)")   # named groups → named columns
s.str.extractall(r"(\d+)")                       # ALL matches → MultiIndex
s.str.findall(r"\d+")                            # all matches → Series of lists
s.str.count("A")
s.str.match(r"^chr\d+$")                         # anchored boolean test
s.str.fullmatch(r"chr\d+")

extract with named groups is the cleanest way to parse a structured identifier:

loc = variants["locus"].str.extract(r"(?P<chrom>chr[\dXYM]+):(?P<pos>\d+)")
loc["pos"] = loc["pos"].astype("int32")

A non-matching row gives NaN, not an error. Always check:

failed = loc["chrom"].isna()
if failed.any():
    print(f"{failed.sum()} unparseable: {variants.loc[failed, 'locus'].head().tolist()}")

split + expand: the workhorse

df[["chrom", "pos", "ref", "alt"]] = df["variant"].str.split("_", expand=True)

expand=True turns one column into several in one line. Two cautions:

• If rows have different numbers of fields, the short ones get None and the column count is set by the longest row.
• The results are all strings. Cast afterwards.

parts = df["variant"].str.split("_", expand=True)
assert parts.shape[1] == 4, f"expected 4 fields, got {parts.shape[1]}"
df[["chrom", "pos", "ref", "alt"]] = parts
df["pos"] = df["pos"].astype("int32")

explode: one row per element

df.assign(csq=df["consequence"].str.split("&")).explode("csq")

A variant annotated missense_variant&splice_region_variant becomes two rows. This is how you get to a countable, groupable form. See Reshaping with pivot and melt.

Bioinformatics parsing recipes

# strip Ensembl version suffixes so IDs will actually join
df["gene_id"] = df["gene_id"].str.split(".").str[0]

# normalise chromosome naming (UCSC ↔ Ensembl)
df["chrom"] = df["chrom"].str.removeprefix("chr")
df["chrom"] = "chr" + df["chrom"].astype(str)

# parse a locus string
df[["chrom", "pos"]] = df["locus"].str.split(":", expand=True)

# parse a GFF attributes column
gff["gene_id"] = gff["attributes"].str.extract(r'gene_id "([^"]+)"')
gff["gene_name"] = gff["attributes"].str.extract(r'gene_name "([^"]+)"')

# parse a VCF INFO field for one key
df["dp"] = df["INFO"].str.extract(r"(?:^|;)DP=(\d+)").astype("Int64")
df["af"] = df["INFO"].str.extract(r"(?:^|;)AF=([\d.eE+-]+)").astype("float32")

# parse a sample FORMAT field
df[["gt", "ad", "dp"]] = df["sample1"].str.split(":", n=2, expand=True)
df[["ref_count", "alt_count"]] = df["ad"].str.split(",", expand=True).astype("int32")

# HGVS-shaped strings: for FILTERING only, never for interpretation
df["accession"] = df["hgvs"].str.split(":").str[0]
df["is_coding"] = df["hgvs"].str.contains(":c.")

# clean sample names from filenames
df["sample"] = df["path"].str.extract(r"([^/]+)\.bam$")

# structured sample IDs
df[["patient", "tissue", "rep"]] = df["sample_id"].str.split("_", expand=True)

The (?:^|;) in the INFO extractions matters: without it, DP= also matches inside MQ_DP= or ExcessHet;DP=. Anchoring to a semicolon or the start of the string prevents that.

On the HGVS line: string manipulation of HGVS is acceptable for coarse filtering (which accession, is it coding). It is never acceptable for extracting positions, comparing variants, or deciding equivalence: a regex cannot know that c.76_78del and c.77_79del are the same event. Use hgvs Parser. See hgvs Pitfalls.

Missing values

s.str.upper()                        # NaN stays NaN, no error
s.fillna("").str.contains("BRCA")    # NaN → False
s.str.contains("BRCA", na=False)     # same, more direct

.str.contains returns NaN for missing input, and NaN in a boolean mask raises when you use it in .loc. Always pass na=False (or na=True) when the result feeds a filter.

Performance

.str methods are vectorised at the pandas level but still involve per-element Python string operations for many methods: they are much faster than .apply(lambda x: ...) but not as fast as numeric ufuncs. With Arrow-backed strings (the pandas 3.0 default) many operations are now compiled and substantially faster.

For very heavy parsing of very large files, consider parsing at read time with a proper format library (pysam, cyvcf2) rather than regex over a DataFrame.

s.str.contains("x", regex=False)     # skip the regex engine when you can

Common mistakes

• NaN propagating into a boolean mask. Pass na=False.
• Forgetting .str is regex by default. s.str.replace(".", "") deletes every character. Pass regex=False or escape it.
• split(expand=True) on ragged rows producing surprise column counts.
• Not casting after splitting. Everything comes back as a string, so pos sorts as "1", "10", "100", "2".
• Unanchored INFO regexes matching the wrong key.
• Silent NaN from a failed extract. Always count them.
• Parsing HGVS with regex for anything semantic.
• Using .apply(lambda x: x.upper()) instead of .str.upper().

See also

pandas dtypes · Reshaping with pivot and melt · Applied - Reading VCF with pandas · Applied - GFF and Genomic Intervals · hgvs Parser · Series

Exercise

Write parse_info(text). Stdin holds one variant per line: an identifier and a VCF INFO field separated by a tab. The INFO field is a ;-separated list of KEY=VALUE pairs, and it may be empty. Return a list of strings.

Use the .str accessor with an anchored regular expression to pull the DP (integer) and AF (float) values out of the whole column at once. The anchor matters: a naive DP=(\d+) also matches inside keys such as MQ_DP= or HAPDP=, so require the key to start at the beginning of the field or immediately after a ;.

Return one id<TAB>dp<TAB>af line per record in input order, with af rounded to three decimal places and the literal NA in either position when that key is absent. Finish with a line missing_dp=<n> counting the records with no DP. A record whose INFO is empty contributes NA for both keys.