Bio.Entrez

Core Principle: Bio.Entrez provides programmatic access to NCBI databases via the E-utilities API, requiring authenticated client identification, rate-limit compliance, and systematic response caching.

Setup: not optional

from Bio import Entrez

Entrez.email = "you@institution.edu"      # REQUIRED by NCBI's terms
Entrez.tool = "my_analysis_pipeline"       # identifies your software
Entrez.api_key = "your_key_here"           # optional; raises the rate limit

NCBI requires an email so they can contact you before blocking your IP if a script misbehaves. Without it Biopython warns and NCBI may throttle you.

Get an API key. It is free from your NCBI account and raises your limit from 3 to 10 requests per second. Store it in an environment variable, not in the script:

import os
Entrez.api_key = os.environ.get("NCBI_API_KEY")

The main calls

Entrez.esearch(db, term, retmax)    # search → list of IDs
Entrez.efetch(db, id, rettype, retmode)   # retrieve records
Entrez.esummary(db, id)             # brief summaries
Entrez.elink(dbfrom, db, id)        # cross-database links
Entrez.einfo(db)                    # what fields does this database have?
Entrez.epost(db, id)                # upload IDs to the history server
Entrez.read(handle)                 # parse XML → nested dicts/lists

Always close the handle, or use a with block:

with Entrez.efetch(db="nucleotide", id="NM_000546.6",
                   rettype="gb", retmode="text") as h:
    rec = SeqIO.read(h, "genbank")

Common patterns

# search then fetch
with Entrez.esearch(db="nucleotide", term="TP53[Gene] AND human[Organism]",
                    retmax=20) as h:
    ids = Entrez.read(h)["IdList"]

with Entrez.efetch(db="nucleotide", id=",".join(ids),
                   rettype="gb", retmode="text") as h:
    records = list(SeqIO.parse(h, "genbank"))

# one accession as GenBank
with Entrez.efetch(db="nucleotide", id="NM_000546.6",
                   rettype="gb", retmode="text") as h:
    rec = SeqIO.read(h, "genbank")

# protein FASTA
with Entrez.efetch(db="protein", id="NP_000537.3",
                   rettype="fasta", retmode="text") as h:
    prot = SeqIO.read(h, "fasta")

# PubMed abstracts
with Entrez.efetch(db="pubmed", id="12345678", rettype="abstract",
                   retmode="text") as h:
    print(h.read())

# gene record as structured XML
with Entrez.efetch(db="gene", id="7157", retmode="xml") as h:
    gene = Entrez.read(h)

# ClinVar for a gene
with Entrez.esearch(db="clinvar", term="TP53[gene] AND pathogenic",
                    retmax=100) as h:
    ids = Entrez.read(h)["IdList"]

db="gene" with id="7157" uses the Entrez Gene ID, not the symbol. Use esearch on db="gene" to go from symbol to ID.

Batching

Never fetch one ID at a time in a loop.

def fetch_batched(ids, db="nucleotide", rettype="fasta", batch=200):
    for i in range(0, len(ids), batch):
        chunk = ids[i:i + batch]
        with Entrez.efetch(db=db, id=",".join(chunk),
                           rettype=rettype, retmode="text") as h:
            yield from SeqIO.parse(h, rettype)
        time.sleep(0.15)

For very large sets, use the history server so you never send the ID list back:

with Entrez.esearch(db="nucleotide", term=query, usehistory="y") as h:
    r = Entrez.read(h)
count, webenv, key = int(r["Count"]), r["WebEnv"], r["QueryKey"]

for start in range(0, count, 500):
    with Entrez.efetch(db="nucleotide", rettype="fasta", retmode="text",
                       retstart=start, retmax=500,
                       webenv=webenv, query_key=key) as h:
        yield from SeqIO.parse(h, "fasta")

Be a good citizen

• ≤3 requests/second without an API key, ≤10 with one. Biopython does some throttling; do not rely on it entirely.
• Bulk downloads outside US peak hours (weekends, or 21:00–06:00 Eastern).
• For anything genuinely large, use the FTP site (ftp.ncbi.nlm.nih.gov) or the datasets CLI. E-utilities is an API for queries, not a bulk transfer mechanism.
• Retry with backoff: the service returns transient 500s under load.

import time
from urllib.error import HTTPError

def robust_fetch(fn, *args, retries=4, **kw):
    for attempt in range(retries):
        try:
            return fn(*args, **kw)
        except HTTPError as e:
            if attempt == retries - 1 or e.code < 500:
                raise
            time.sleep(2 ** attempt)

Cache aggressively

Network calls are slow, rate-limited, and not reproducible: the record you fetch today may differ from the one you fetch next year, because sequences get revised.

from pathlib import Path

def cached_genbank(acc, cache_dir="ncbi_cache"):
    cache = Path(cache_dir); cache.mkdir(exist_ok=True)
    path = cache / f"{acc}.gb"
    if not path.exists():
        with Entrez.efetch(db="nucleotide", id=acc,
                           rettype="gb", retmode="text") as h:
            path.write_text(h.read())
    return SeqIO.read(path, "genbank")

Cache the accession with its version (NM_000546.6, not NM_000546). An unversioned accession returns whatever is current, so your pipeline's behaviour changes silently when NCBI updates the record. This is the same reproducibility principle as pinning your UTA snapshot: see UTA and Data Providers.

Security note

Entrez.read parses XML. Biopython 1.87 fixed CVE-2025-68463 in Bio.Entrez.Parser. Keep Biopython up to date, especially if you parse XML from anywhere other than NCBI itself.

Bioinformatics examples

# all RefSeq transcripts for a gene
with Entrez.esearch(db="nucleotide",
                    term="TP53[Gene] AND human[Organism] AND refseq[Filter] AND mRNA[Filter]",
                    retmax=50) as h:
    ids = Entrez.read(h)["IdList"]

# gene → protein via elink
with Entrez.elink(dbfrom="gene", db="protein", id="7157") as h:
    links = Entrez.read(h)
protein_ids = [l["Id"] for l in links[0]["LinkSetDb"][0]["Link"]]

# taxonomy lineage
with Entrez.efetch(db="taxonomy", id="9606", retmode="xml") as h:
    tax = Entrez.read(h)
print(tax[0]["Lineage"])

# assembly summary
with Entrez.esummary(db="assembly", id="GCF_000001405.40") as h:
    print(Entrez.read(h))

# build a local reference set for a gene panel
for gene in PANEL:
    with Entrez.esearch(db="nucleotide",
                        term=f"{gene}[Gene] AND human[Organism] AND refseq_select[Filter]") as h:
        ids = Entrez.read(h)["IdList"]
    if ids:
        rec = cached_genbank(ids[0])
        SeqIO.write(rec, f"refs/{gene}.gb", "genbank")
    time.sleep(0.15)

refseq_select[Filter] restricts to the single representative transcript per gene, which is usually what you want for a panel.

Alternatives

Need	Better tool
Bulk genome/annotation download	NCBI datasets CLI, or FTP
Ensembl data	Ensembl REST API, or pyensembl
UniProt	UniProt REST API
Variant annotation	Ensembl VEP, ClinVar VCF downloads
Transcript–genome alignments	UTA: see UTA and Data Providers
Sequences by accession	SeqRepo: see SeqRepo

For variant work specifically, SeqRepo and UTA are far better than Entrez: local, versioned, fast, and reproducible.

Common mistakes

• Not setting Entrez.email.
• No API key, so you are limited to 3 req/s.
• One request per ID in a loop.
• No caching, making the pipeline slow and irreproducible.
• Unversioned accessions.
• No retry logic for transient 500s.
• Bulk downloading through E-utilities instead of FTP.
• Not closing handles.
• Hardcoding the API key in a committed script.

See also

SeqIO · SeqRecord · Bio.Blast · SeqRepo · UTA and Data Providers · Applied - Reproducible Environment

Exercise

Entrez.read turns an E-utilities XML response into nested dictionaries, and efetch then wants those ids sent back in batches rather than one at a time. Biopython is not installed here, so parse the response with xml.etree.ElementTree from the standard library and build the batches yourself: the network is never touched.

Write id_batches(xml_text, batch), taking an eSearchResult document and a batch size, and returning the list of output lines. Read the reported total from the Count element, using 0 if it is absent. Collect the ids from Id elements inside IdList only: an E-utilities response can carry other Id elements elsewhere in the tree, and a naive full-tree walk picks those up too. Drop duplicates while preserving first-seen order. Print count=<reported> unique=<n>, then one line per batch as batch<k>=<comma-joined ids> numbered from 1. When IdList is empty, print the no ids line instead of any batch line. The batch size arrives on the first line of input and the XML document follows.