| ip_input | route_asn | ipinfo_asn | route_ipinfo_asn_agree | peeringdb_name | peeringdb_info_types | education_network_reported | enterprise_network_reported | cable_dsl_isp_network_reported | cloud_matches | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 45.88.80.1 | 786 | 786 | True | Janet | (Educational/Research,) | True | False | False | () |
| 1 | 1.44.96.1 | 16509 | 16509 | True | Amazon.com | (Enterprise,) | False | True | False | () |
| 2 | 52.95.110.1 | 16509 | 16509 | True | Amazon.com | (Enterprise,) | False | True | False | (aws,) |
| 3 | 8.8.8.8 | 15169 | 15169 | True | Google LLC | (Content,) | False | False | False | (googleservices,) |
| 4 | 81.2.69.142 | 20712 | 20712 | True | Andrews & Arnold Ltd | (Cable/DSL/ISP,) | False | False | True | () |
| 5 | 2606:4700:4700::1111 | <NA> | 13335 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | (cloudflare,) |
| 6 | 2001:db8::1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | () |
| 7 | ::ffff:34.82.1.5 | 396982 | 396982 | True | Google Cloud | () | False | False | False | (googlecloud,) |
| 8 | 1.2.3.4:443 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> |
| 9 | not an ip | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> |
| 10 | None | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> |
The question
An application log often starts with little more than a source IP address. Public data can add useful context: the routed prefix, autonomous system number (ASN), network name, broad geography, cloud-provider range, known VPN or datacentre range, and whether the address appeared in a Tor exit list.
None of those fields answers the question “is this request abusive?” They describe infrastructure. The defensible output is a set of dated observations with provenance, not one all-purpose bad_ip label.
TipThe compact rule
Preserve each source as its own feature, record when it was observed, and let later analysis decide how much weight it deserves.
Five different kinds of information
The exploratory work for this page tested routing tables, ASN metadata, a multi-provider range aggregator, Tor exits, VPN/datacentre lists, and a hand-maintained ASN blocklist. It did not download AWS and Google Cloud’s first-party feeds directly. These sources are easy to join technically and easy to conflate conceptually.
| Layer | Example output | What a match supports | What it does not support |
|---|---|---|---|
| Route origin | asn, routed prefix |
The ASN originating the best matching prefix in that snapshot | The end user, business type, or intent |
| Organisation metadata | network name, website, self-described type | Context about the network operator | An independently verified residential/hosting classification |
| Provider-published ranges | provider, service, region | The address fell inside a range the provider published for that purpose | Complete coverage of the provider or proof that a customer generated the request |
| Observed service list | tor_exit_listed |
The address appeared in the relevant Tor list at the observation time | Abuse, identity, or historical Tor use outside that snapshot |
| Curated heuristic | vpn_listed, datacentre_listed, hosting_asn_listed |
A particular maintainer classified the range or ASN that way | Ground truth, completeness, or a safe automatic block decision |
The distinction matters because the rows have different units. A Tor list is largely an IP-level observation. A provider feed is a set of prefixes. PeeringDB describes a network associated with an ASN. A hand-built “hosting ASN” list applies one label to every routed address associated with that ASN. Collapsing them into one boolean discards both provenance and uncertainty.
A source hierarchy
1. Start with routing, not reputation
An IP-to-ASN table answers the narrowest useful question: which ASN originated the longest matching routed prefix in a particular routing snapshot? It supplies a join key for network-level metadata. It does not establish that the ASN owns every address, operates the end host, or belongs to a particular business category.
Routing data is time-varying. Prefixes are announced, withdrawn, transferred, and sometimes originated by different ASNs. Historical event analysis therefore needs a routing snapshot close to the event, not only the current table. One suitable source is CAIDA’s RouteViews Prefix-to-AS mappings, produced daily for IPv4 and IPv6 with archives extending back to 2005 and 2007 respectively. Its Public AUA requires citation and asks users to report publications—including web pages—that use the data.
2. Add network metadata without promoting it to truth
PeeringDB exposes network names, websites, reach, traffic characteristics, and info_types through its API. These records are valuable explanatory metadata, but many of the fields are self-described and coverage is not universal. The enrichment retains all reported types and provides nullable conveniences for Educational/Research, Enterprise, and Cable/DSL/ISP. If an active record has a known type list that omits one of those labels, the corresponding field is False; if the record or its type list is unavailable, the field is NA. Neither case establishes who uses an individual address, whether it is shared, or whether its traffic is safe.
The local Stanford ASdb artifact named 2026-03_categorized_ases.csv is also joined by normalized route ASN. The filename identifies the local artifact but is not treated as evidence of an exact publication date. Its wide CSV supplies up to 68 industry category/sub-category pairs per ASN; it does not contain a separate organisation-name field. Every supplied pair is retained under asdb_ names. These inferred categories remain separate from PeeringDB’s self-reported types, and disagreement is not silently reconciled.
IPinfo Lite supplies a separate CIDR and single-address lookup for country, continent, ASN, AS name, and domain. Its ASN is normalized from values such as AS13335, but it never overwrites route_asn or asn_country. ipinfo_country is IP-range geolocation, not the country of the ASN organisation. Agreement is reported only when both sources return an ASN.
Likewise, an ASN category from an inferred classification model should retain the model name and version. The former CAIDA AS Classification dataset, for example, trained on PeeringDB-derived ground truth supplemented by 500 manually classified enterprise networks. Its features included AS relationships, customer-cone size, advertised IPv4 space, popular domains mapped to the AS, and observed address activity. CAIDA reports a positive predictive value of 70% and now marks the dataset as unsupported. That is useful research context, not a timeless label.
3. Prefer first-party cloud ranges
Provider feeds are strong evidence for the specific claim their publisher makes:
- AWS publishes
ip-ranges.json, with service and region fields. AWS explicitly says the file omits some services and customer-owned BYOIP space. - Google Cloud publishes
cloud.json, covering global and regional external ranges available to customer resources.
These should be stored as separate provider/service/region matches. The notebook uses rezmoss’s third-party aggregation: its normalized table carries provider, service, region, and last_updated fields assembled from multiple upstream sources. cloud_matches remains the provider-only compatibility field; cloud_match_details retains the matching CIDR and those related attributes as one record. Those fields were not independently reproduced from the AWS and Google feeds in this exploration. An aggregate inherits every upstream definition, refresh schedule, overlap, and omission, so keep the aggregator, original publisher, and source URL distinct wherever possible.
Overlaps are not an error to erase blindly. A broad parent prefix may carry one provider label while a more specific prefix carries another. The provider enrichment returns every applicable record, removes exact duplicates, and orders the result deterministically; it does not discard a broader match merely because a more specific range also matched. Provider region describes the published infrastructure range, not necessarily the user’s location. A documented provider record may itself name a relay service, but the implementation does not infer relay status from an arbitrary cloud match. This is distinct from route-origin semantics.
4. Treat Tor as a dated observation
The notebook downloaded the exact endpoint https://check.torproject.org/torbulkexitlist. The Tor Project describes this default bulk artifact as addresses it has observed being used by exit relays through active measurements. Since its 2020 service change, bulk-list queries are no longer filtered by exit policy: an observed exit address is returned regardless of query parameters. It should therefore be described as a list of observed Tor exit addresses, not as proof that every listed address can reach a particular destination. Raw measurements are archived by CollecTor. See the Tor Project’s explanation of the exit-list service change.
A present-day tor_exit_listed=True match is an observation tied to that exact artifact; it cannot establish that an address was a Tor exit when an older login occurred. If the local artifact is unavailable, the notebook leaves tor_exit_listed as NA; it does not turn an unavailable source into an empty list or a checked non-match.
Do not silently substitute another Tor artifact. Different list-building methods have produced different counts, and this exploration did not compare them. Record the endpoint, destination assumptions, retrieval time, and hash with the result.
The 25 October 2026 rerun of that exact endpoint contained 1,213 distinct addresses. This is a reproducibility detail, not a stable size estimate: the live list can change after the snapshot.
Most importantly, a Tor match identifies shared privacy infrastructure. It does not identify a person and it is not evidence of abuse.
5. Keep community lists visibly heuristic
Community VPN, datacentre, and hosting-ASN lists can add coverage that first-party feeds do not provide. They can also contain stale ownership, broad labels, false positives, and undocumented judgement calls.
The explored bad-asn-list is a useful example. It is an open defensive artifact accompanied by one operator’s account of using hosting-ASN blocking against signup abuse. It is not a measured reference classification: the reported effect is single-site and anecdotal, the selection method is not documented as a reproducible study, and one ASN-level decision can affect mixed residential, enterprise, and hosted traffic. In this project it is retained as a heuristic feature and historical operator account (SRC-052), not adopted as ground truth.
The same rule applies to merged range repositories: useful for comparison and recall experiments, but label the aggregator and snapshot rather than implying that every entry is an official provider assertion.
A defensible output schema
Avoid a single is_proxy or is_bad column. The companion implementation’s enrich() call returns source-specific fields with explicit missingness:
| Field | Meaning | Source | Missing-value semantics | Main limitation |
|---|---|---|---|---|
ip_input, ip_normalized, ip_version |
Original value, canonical address, and family | Input/parser | Parse failures retain ip_input; normalized fields are NA |
IPv4-mapped IPv6 is deliberately normalized to IPv4 |
route_asn, asn_name, asn_country |
Origin ASN and route-table descriptions | IPtoASN | NA for invalid/IPv6/ASN 0/no route/absent source |
Route origin is not user identity; asn_country is not IP geolocation |
ipinfo_country, ipinfo_continent |
IP-range geolocation labels | IPinfo Lite | NA for no matching row/source |
Approximate range geography, not the user’s location or ASN country |
ipinfo_asn, ipinfo_as_name, ipinfo_as_domain |
Independent ASN assignment and AS metadata | IPinfo Lite | Individual missing fields remain NA |
Never overwrites route metadata |
route_ipinfo_asn_agree |
Equality of normalized route and IPinfo ASNs | Derived comparison | NA unless both ASNs exist |
A disagreement does not say which source is right |
peeringdb_name, peeringdb_website, peeringdb_info_scope, peeringdb_updated |
Active network-record metadata | PeeringDB on route_asn |
NA for no record or missing field/source |
Community-maintained organisational context |
peeringdb_info_types |
Every self-reported network type | PeeringDB | NA for no record/unavailable types; empty tuple for a known empty list |
Self-reported, non-exclusive, incomplete |
education_network_reported, enterprise_network_reported, cable_dsl_isp_network_reported |
Membership in three reported PeeringDB categories | PeeringDB | NA if record/types unavailable; otherwise nullable Boolean |
Not a user, safety, usage-volume, or confirmed-NAT label |
cloud_matches |
Compatible tuple of every matching provider name | rezmoss aggregate | Empty tuple means checked/no match; NA means unsupported/unavailable |
Third-party aggregation; overlaps retained |
cloud_match_details |
Matching CIDR/provider/service/region/source-update records | rezmoss aggregate | Empty tuple means checked/no match; NA means unsupported/unavailable |
Region describes the published range; no generic relay inference |
asdb_category_pairs, asdb_layer1_categories, asdb_layer2_categories |
All inferred industry category pairs and convenience projections | Stanford ASdb on route_asn |
NA for no ASN row/source |
Not reconciled with PeeringDB or promoted to verified fact |
tor_exit_listed |
Exact address in the dated Tor artifact | Tor bulk list | NA when family/source unavailable |
Dated shared-infrastructure observation, not abuse evidence |
vpn_listed, datacentre_listed |
Separate community-range matches | X4BNet | NA for IPv6/invalid/absent source |
Broad, dependent heuristics |
hosting_asn_listed |
Route ASN in the curated list | bad-asn-list | NA without route/source |
Maintainer heuristic, not ground truth |
Source URL, genuine retrieval time when known, publisher update time, filesystem modification time, and SHA-256 live in a companion source-level manifest rather than being repeated on every event row.
Use nullable values deliberately. False should mean the address was checkable against the source and no match was found. NA should mean the lookup was not applicable or could not be performed—for example, applying an IPv4-only list to IPv6. Unknown ASN metadata should remain unknown; it is not adverse evidence.
Worked implementation
The reusable functions are in src/openbotrisk/ip_enrichment.py, with boundary and semantics checks in tests/test_ip_enrichment.py. The executed demo notebook loads named local snapshots without network access, measures coverage and disagreement, and calls enrich() on deliberately awkward inputs.
The site build embeds the notebook’s committed output; it does not download live data. Run the notebook explicitly when creating a new dated snapshot, inspect the output, and commit the notebook with its outputs. The main enrichment call is:
The real examples include 45.88.80.1, whose route ASN maps to Janet/Jisc (AS786) and whose PeeringDB record reports Educational/Research; 81.2.69.142, whose route ASN maps to Andrews & Arnold (AS20712) and reports Cable/DSL/ISP; and 1.44.96.1, whose route ASN maps to Amazon.com (AS16509) and reports Enterprise. Educational classification supplies network context, not evidence about how many users share one IP. Amazon also demonstrates why PeeringDB Enterprise is not a reliable shared-office indicator. None of these types attributes a user, confirms NAT, or verifies that an unmatched address is safe or residential. An empty provider tuple means the supported address family was checked and did not match. Missing values mean that the source was absent or inapplicable. Report that missingness alongside any flag rate:
CautionCurrent implementation limits
IPinfo and the detailed rezmoss prefix index support IPv4 and IPv6 without expanding networks. The IPtoASN route table and X4BNet interval lists remain IPv4-only, so their fields are NA for ordinary IPv6. PeeringDB and ASdb currently depend on the route_asn join; consequently they may remain unavailable for IPv6 even when IPinfo supplies an independent ASN. This page deliberately does not add an IPinfo-ASN fallback or change that lookup policy. The pipeline loads these public snapshots in memory and is intended for reproducible investigation, not an unbounded production stream.
Source inventory for the executed run
The inventory is generated from the notebook’s shared provenance records rather than a separately maintained list. It includes every input used in the run, including Tor and the community lists. Retrieval time, source publication or generation time, and filesystem modification time remain separate. A manifest retrieval time is used only when its recorded SHA-256 matches the local artifact:
All requested datasets were present in the executed run. The local IPtoASN, IPinfo and ASdb files still have no evidenced retrieval timestamp; the ASdb artifact also has no artifact-linked publication/update date or organisation-name field in the inspected CSV. Those values remain unknown. First-party AWS and Google feeds remain a deferred comparison; the current run uses the existing rezmoss aggregate.
Classification coverage and ASN disagreement
Record matching is not the same as useful classification coverage. The table separates absent source records, present records with missing or empty classification, and records with at least one usable classification. Its UK row is precisely the distinct non-zero ASNs attached to an IPtoASN route row whose source-defined country field is GB; it is not all networks serving UK users:
Across all 79,310 route ASNs, PeeringDB supplied usable info_types for 26.79%; 63.08% had no PeeringDB record and 10.14% had a record without a usable type. In the 2,072-ASN IPtoASN GB subset, usable PeeringDB classification was 35.57%, while ASdb supplied usable categories for 94.50%. These are ASN-level diagnostics, not traffic-, user-, or address-space-weighted estimates.
The IP-level comparison uses the same fixed-seed random sampling method across the full eligible IPv4 source-row set for both the overall and IPinfo country_code=GB views:
In the 5,000-row IPinfo GB sample, 4,366 rows supplied both ASNs; 21 disagreed, or 0.48% of comparable rows. PeeringDB supplied a usable classification for 73.74% of all sampled rows and ASdb for 86.76%; these denominators include the 12.62% without a route ASN, for which route-joined classifications could not be attempted. IPinfo country here is IP-range geolocation, not ASN country. The disagreement count is a prompt to inspect snapshot timing, range specificity, and source methodology, not evidence that either ASN source is generally wrong.
The identifiers in the synchrony experiment, such as ip_res_000607, are deliberately synthetic and are not submitted to these public lookup services. A separate authored fixture in notebooks/methodology_synchrony.ipynb demonstrates how network context can be joined to those scenario identifiers. It is labelled as scenario data and does not measure public-list accuracy or real-world detection performance.
Lookup mechanics that change the result
Provider overlap and route-origin semantics
IP ranges are nested, so a lexical join is insufficient, but the correct policy depends on the source. Provider enrichment returns all matching detailed records so overlaps remain visible. IPinfo contains both CIDRs and single-address rows, so its index applies longest-prefix matching. Raw routing prefixes generally require source-defined best-match semantics; the IPtoASN input used by the companion implementation is already a resolved, non-overlapping address-range table, so the code looks up its containing interval rather than inventing another longest-prefix rule.
IPv4 and IPv6 are separate paths
Converting IPv4 to a 32-bit integer gives a fast interval lookup, but that implementation does not generalise unchanged to IPv6. The IPinfo and provider indexes use integer prefixes for both families; IPv4-mapped IPv6 addresses are deliberately normalized to IPv4. The route table and community interval lists remain IPv4-only. An IPv6 address presented to one of those sources produces NA, not False.
Snapshot before transform
For every mutable download:
- save the original bytes under a dated path;
- record the URL, actual retrieval time, publisher update time if supplied, and SHA-256;
- parse into a derived table without modifying the raw artifact;
- version the lookup policy as well as the data;
- retain the snapshot needed to reproduce an event-time result.
Filesystem modification time is recorded separately when useful. It is not a substitute for retrieval time: a copied local file with no retrieval record keeps retrieved_at missing.
This is especially important for Tor exits, BGP-derived route origins, and cloud feeds, all of which change over time.
Test meaning, not just code
A lookup implementation needs boundary cases: first and last address in a prefix, addresses immediately outside it, overlapping prefixes, malformed input, private and reserved space, IPv4-mapped IPv6, ordinary IPv6, and duplicate source rows. It also needs semantic checks: a missing source record must not become a risk flag, and a match must not become an abuse label.
Dated coverage measurements
The executed notebook calculates these figures from the available, hash-recorded local artifacts. “Addresses” counts the union represented by each internally non-overlapping IPv4 CIDR file; it is not a count of observed users or active hosts. If a source is missing, its row is explicitly unavailable rather than zero:
Expanding the X4BNet datacentre ranges to one dotted IPv4 address per line would produce about 5.55 GB of text (5.17 GiB), calculated from the ranges without materialising that file. Range lookup avoids that expansion.
The lists are not independent. X4BNet defines its datacentre output as VPNs and datacentres, while its VPN output is intended to be the narrower list. The overlap calculation is also guarded against missing inputs and empty denominators:
In this run:
- 3,145,854 of 3,170,072 VPN addresses (99.24%) were also covered by the X4BNet datacentre list;
- 8,249 of 10,931 VPN network starts fell inside a datacentre network, and 8,242 whole VPN networks were contained there;
- 234,191,767 of the 255,279,702 rezmoss merged IPv4 addresses (91.74%) were covered by X4BNet datacentre.
The address-intersection figures are exact for these three normalized snapshots. The 8,249 start-point figure is retained only as a rough network-level diagnostic; it is not evidence that one file is a strict CIDR-level superset of the other. Agreement analysis must account for this construction instead of treating the lists as independent votes.
Other profiles help explain the source limitations:
| Artifact | Snapshot result | Interpretation |
|---|---|---|
rezmoss all_providers.json |
466,400 rows; 299,342 IPv6 (64.18%) | Row count is not address-space coverage; IPv4 and IPv6 need separate lookup paths |
| IPinfo Lite | 4,275,771 network/single-address rows; 835,909 missing ASN values | Row count is not address-space coverage; longest-prefix matching preserves more-specific rows |
| PeeringDB active networks | 35,735 rows; 10,370 known empty info_types lists |
A known empty list supports checked category False; an absent record still remains NA |
| Stanford ASdb | 115,490 ASN rows; 137 columns | Multiple category pairs apply and their ordering has no significance |
bad-asn-list.csv |
742 rows; 723 unique ASNs | Duplicate ASNs must be removed before matching |
bad-asn-list.csv, 16-bit ASN |
641 rows; 623 unique ASNs | The older-ASN skew is consistent with dated coverage, but does not by itself measure freshness or recall |
NoteSnapshot provenance and hashes
The provenance evidence and source manifest are recorded in notebooks/public_ip_enrichment_demo.ipynb; the earlier exploratory audit remains in notebooks/temp_ips.ipynb and notebooks/ip_enrichment_snapshot.py. The notebook keeps genuine hash-matched manifest retrieval times, evidenced publisher generation/update times, and filesystem modification times separate, and leaves unknown values missing. No new raw dataset is added as a site asset or branch change.
What the exploration established
The notebook proved that the source families can be downloaded, parsed, and joined locally. It also exposed the design decisions that need to precede a reusable pipeline:
- a current Tor list is simple to match but inappropriate for historical claims;
- the rezmoss aggregate carries provider, service, and region fields from its upstream inputs, but this exploration did not validate them against the first-party feeds;
- multi-provider aggregates contain IPv4, IPv6, and overlapping entries, so row counts are not address-space coverage;
- PeeringDB adds readable organisational context but uses self-reported types and incomplete coverage;
- broad VPN/datacentre lists cover hundreds of millions of addresses and should be stored as ranges, not expanded into individual IP addresses;
- a curated hosting-ASN list is a heuristic whose incremental coverage and false-positive exposure should be measured before use.
This produces design guidance and coverage measurements, not a predictive finding. Building a universal “bad infrastructure” detector from these sources would overstate what the data can support.
What to measure next
The next useful experiment is comparative, not additive. Given a dated set of login IPs:
- enrich it with route origin and first-party provider/Tor snapshots;
- apply each community range or ASN heuristic separately;
- measure agreement, unique additions, missingness, and stability across snapshots;
- inspect disagreements by network type and prefix size;
- only then test whether any feature adds predictive value against an independently obtained outcome label.
Without the last step, the output remains infrastructure description. That is still useful—as long as it is named honestly.
Sources used on this page
| Source | Use here | Licence or access note checked for this draft |
|---|---|---|
X4BNet lists_vpn |
VPN and combined VPN/datacentre IPv4 lists | Repository says the scripts, source files, and generated output are MIT licensed |
rezmoss cloud-provider-ip-addresses |
Unified provider table and merged IPv4 comparison list | Repository is dedicated under CC0 1.0; that does not remove any separate conditions attached to upstream feeds |
| IPtoASN | Exploratory IP-to-ASN and country table | Site labels the downloadable database PDDL 1.0 |
| Stanford ASdb | Local multi-label ASN industry-category artifact | Public research download; no artifact-linked publication date or explicit dataset licence is recorded here, so further raw redistribution is not assumed |
| IPinfo Lite | CIDR country/continent and independent ASN metadata | CC BY-SA 4.0 with attribution to IPinfo; adaptations distributed from it must preserve the share-alike terms |
| PeeringDB | Network metadata and self-described info_types |
The AUP permits Internet research and analysis but restricts bulk onward distribution; the pre-existing local snapshot is not copied into site assets |
| CAIDA RouteViews Prefix-to-AS | Historical routing source | Public AUA requires citation and publication reporting |
| CAIDA AS Classification | Legacy classifier example | Unsupported legacy dataset; no download access |
| Tor Project exit-list service | Interpretation of the default bulk list and CollecTor pointer | Exact notebook endpoint named in the text; unavailable retrieval remains NA |
Hamachek bad-asn-list |
Curated hosting-ASN heuristic (SRC-052) |
MIT; treated as an anecdotal defensive artifact, not ground truth |
| AWS IP ranges | First-party alternative not tested in this run | Documentation says ip-ranges.json omits some services and BYOIP ranges |
| Google Cloud IP ranges | First-party alternative not tested in this run | Documentation describes cloud.json as customer-usable global and regional external ranges |
Related reading