transitio.fetch

transitio.fetch(aoi=None, when=None, *, place=None, tiers=None, exclude=None, on_unknown='include', on_untrusted_selector='auto', contained='drop', feeds=None, index=None, credentials=None, modes=None, duplicate_trips='drop', expired='skip', repair=False, crop=True, osm=True, refresh_token=None, cache_dir=None, directory=None, country_code=None, use_cache=True, progress=True, **budgets)

Fetch everything cafein needs for an area in one call.

Pass exactly one of aoi (a geometry, bbox or place name geocoded once) or place (a place name, QID or Place). With place, feeds are selected from the feed index by tier – tiers, exclude and on_unknown filter the edges – and the place geometry supplies the AOI. With aoi, the feeds come from the feed index too when its places cover at least half of the area’s land (transitio.index.area(), country_code keeping one country’s places): those of the area’s places, selected as for place (feeds()), and FetchResult.places lists the places. When no index is installed, the index has no places or they cover less than half of the area, the call falls back to the catalogue path: the Mobility Database catalogue is searched for feeds whose bounding box meets the area’s, and before any feed or OSM download a UserWarning gives the reason and the number of feeds found, e.g. "the feed index's places cover 20% of the area; 3 feeds from the Mobility Database catalogue by bounding box", and without an installed index how to install one. index=False searches the catalogue without a warning; it is refused with place. tiers, exclude, on_unknown, on_untrusted_selector, contained, feeds and credentials apply on both index paths; on the catalogue path, tiers, exclude, on_unknown="exclude", another on_untrusted_selector or contained than the default, feeds and credentials raise ValueError naming the option and the reason, before anything is downloaded. country_code applies only with aoi. When a selector cannot be trusted – its evidence was missing at build time, or its fingerprint no longer matches the download – on_untrusted_selector decides the outcome: "auto" (default) skips the feed when an exclude was asked for and otherwise delivers it whole with its tier treated as unknown; "whole" always delivers it whole; "drop" always skips it; "error" raises StaleSelectorError. A schema-10 index records the larger feeds whose stops and routes contain a feed’s (IndexedFeed.contained_in). With contained="drop" (default) containers are processed first, and a contained feed is left out before download ("contained in <id>") when a container was delivered whole (not cut to a route selection) or skipped as the same content as a feed delivered whole, and conditional HEAD probes (as for expired) prove both archives unchanged since indexed: the container’s, sent before its download, and the contained feed’s. A container downloaded as a catalogued dataset proves nothing. Otherwise the feed is processed as usual and its note says why: "kept: containment in <ids> not proven current", "kept: container <id> skipped" or "kept: container <id> cropped to selected routes". FetchResult.contained maps each feed left out as contained to the delivered feeds that carry it, in selection order: its containers, each one skipped as the same content as a feed delivered whole standing for that feed, and, with a container that lost repeated trips (below), the feeds holding them; a feed with no such feed delivered is not listed. contained="keep" leaves no feed out for containment, and contained is empty.

Resolves and crops the OSM extract, discovers the GTFS feeds (the indexed feeds of the place or the area, or the catalogue’s overlapping the AOI), downloads each feed, spatially crops it, optionally repairs it, validates it, and builds a merged report per feed. With an API token, downloads come from catalogued dataset versions (checksum-verified, with the hosted canonical-validator report); without one, the unversioned latest hosted zip is fetched — a moving target with no upstream checksum, documented in its provenance sidecar as such. On the index paths, a feed without a catalogued dataset, or whose dataset download fails, is read from the first of its indexed URLs that serves a zip archive: the Mobility Database direct download, the Transitland Atlas static feed, then the Mobility Database hosted copy; on a schema-11 index the URL the index crawled (IndexedFeed.download_url) comes first and the hosted copy second. A URL fragment names the member of the archive the feed is read from, a nested zip (.../gtfs.zip#1/google_transit.zip) or a folder; each such archive is downloaded once per call, and the feed’s sidecar records its archive_url and archive_sha256. Each download is kept in the cache as a version of its feed, <cache_dir>/gtfs/<feed folder>/<sha256>.zip beside a .provenance.json sidecar recording every acquisition of those bytes; a download identical to a cached version keeps that version. A report’s provenance describes the version’s first acquisition: its source_url, retrieved_at, fetched_from and download_errors, with sha256, feed_id and, for a catalogued dataset, dataset_id and service_date_range.

On the index paths a cached version that serves the request is used without a download. A feed’s versions are tried before any dataset selection, probe or download: first the one that served the same request before, then the newest first, each checked as a download is (route selector, crop, validation, the day checks). One whose route selector no longer matches the index is passed over; when nothing else serves and no download succeeds, on_untrusted_selector decides on the newest such one. A day no cached version serves downloads the feed, keeping the older versions, so a repeated call delivers what it delivered before, offline too. A warm cache can decide differently from a cold one: a version that serves the day is used without the expired probe, and a cached version counts as unchanged since indexed for contained only when a probe proved it so under the index snapshot in use. A protected feed without usable credentials uses only versions once fetched without them; credentials for a provider count alike whichever key they hold. The hosted validation report of a dataset is stored with it at its first use, so a reused dataset is reported as when downloaded. The catalogue path reuses its feeds’ versions alike, though the Mobility Database is still searched for the feeds; a dataset is selected only for a feed no cached version serves. What the crop, repair and validation make of a cached version is stored with it, keyed by the transitio release, the exact area when cropped to it, the routes, crop, repair and the budgets, and a later call making the same reads it back; the mode filter, the day checks and the report run again on every call, and deleting a version deletes what was made of it. Every overlapping feed is processed, in a deterministic order with official feeds first; one broken feed never aborts the others — it lands in skipped with its reason. A feed lacking a file GTFS requires is skipped, with or without when and repair: "missing required file agency.txt" (several names sorted, "missing required files ...") or "missing calendar.txt and calendar_dates.txt", joined with "; " when both apply; a crop that keeps no trip leaves both calendar files out. A download whose content equals a feed already delivered in the call is skipped as "same content as <feed id>" when its routes are within those delivered from that archive (a feed delivered whole carries all); otherwise it is delivered cut to its own routes, same_as naming the earlier feed.

On the index paths, delivered feeds whose route keys (agency name, route short else long name, mode) and stops (coordinates at 3 decimals) share 0.9 and 0.8 or more are versions, ranked by later start, more trips, then candidate order. Agency names compare casefolded, without diacritics, punctuation or a trailing legal form (Ltd, Oy, S.A. and the like). A feed with at most one agency row, none named, whose routes name at most one agency_id is unnamed, and its pairs compare routes by name and mode only. With when, one is left out as "another version of <id>" when a kept version pairs with it and kept versions run, by trip signature (which leaves out the agency), every trip it runs on the day, a headway trip matching one with the same stops, relative times and frequency rows; the top version and the containers a left-out feed relied on stay, as does one with transfers or pathways. With when, a feed whose trips running on the day all belong to services with a calendar.txt row spanning 4,000 days or more is undated, noted "placeholder calendar <start> to <end>", the widest dates of those rows, and ranks by its placeholder start. An undated feed also pairs on stops alone with a dated one starting after its placeholder start, so its version_of route overlap may be under 0.9, and a kept such version leaves it out whatever it runs on the day and its transfers or pathways; the top version and the containers relied on still stay. A left-out version’s fares are not delivered. Without when none is left out; similar feeds are noted. A feed whose routes, stops or, with when, calendars cannot be read, or with a blank agency name that is not unnamed, is never a version.

With duplicate_trips="drop" (default) the delivered feeds do not repeat each other’s trips. A trip that a feed earlier in the selection record also runs is left out of the later feed: on the index paths the record follows the view (feeds() or feeds(): category, then relevance, then id), on the catalogue path the order above. Trips compare as merge_feeds() compares them: route name and mode (the route type’s basic mode, as modes names them, else the type itself, so local bus 704 equals bus 3), stops and times, nearly (within 50 m and 3 minutes; with "exact" only exactly), pickup and drop-off, and frequencies, whatever the agency. "keep" compares nothing and delivers every feed with all its trips. With when the trips running that day are compared, each earlier trip covering one later trip; without it a trip is left out only when it is covered on every date it runs, which reads the whole calendar and, on a large network, takes longer and more memory than a study day. A trip whose services cannot be read or are not declared, that cannot be signed or that transfers.txt names is never left out. A feed declaring a time zone not equivalent to the others’ (as merge_feeds(timezones="skip") decides), over max_total_bytes or unreadable is not compared. A feed that loses trips is delivered cropped without them (crop_feed()’s exclude_trips), so the stops, shapes, calendars, routes, transfers and pathways that only those trips used go too; with crop=False and no route selection this is the feed’s first crop. Delivered feeds are separate feeds, so no transfer or pathway links them. The cut feed is validated again, its report (listing also the rows this crop left out) and feed_window come from it, and its note says "<n> repeated trips of <ids> left out". A feed whose every trip in scope repeats is skipped as "every trip repeats a trip of <ids>" (with when, "every trip on <day> repeats a trip of <ids>"), and with modes a feed left without a requested mode as "serves [...] after repeated trips were left out, not [...]", the later feeds then compared again without it; the feed_window of either is that of what it kept. A step that fails, a crop that would also leave out trips repeating nothing, or a cut feed lacking a file GTFS requires keeps the feed’s trips, noted "repeated trips kept: <error>". What was found is stored with each feed’s cached version, keyed by the delivered feeds and their outputs, duplicate_trips, the day and modes, and a later call comparing the same reads it back.

Parameters:

• aoi (geometry, GeoDataFrame/GeoSeries, tuple or str) – Area of interest (place names are geocoded via Nominatim once, and the resulting geometry drives every stage).

• when (str or datetime.date, optional) – Study day the feeds must run on, YYYY-MM-DD. Dataset-version selection needs an API token; with or without one, a feed is skipped when its computed service window (the outer bounds of actual calendar activity, not the published range) ends before the day ("service ended <end>") or starts after it ("service starts <start>, after <day>"), or when its validation report for the day counts no active trip and carries the no_service_on_reference_date notice ("no service on <day>"). Without when there is no study day, only today: a feed is skipped only when its computed window ended before today, and one that starts later or runs on other weekdays stays. An unknown window passes the window checks, and a report without a moment for the day passes the day check.

• feeds (list of str, Recommendation or object with feed_ids, optional) – Fetch only these feeds of the index: a list of feed ids (or one id), a Recommendation, whose place and day the call then takes (fetch(feeds=place.recommend(day)); another place, any aoi or a when on another day raises ValueError), or an object whose feed_ids attribute lists them. They are taken from every relevance category of the place, or of the area’s places, not only the default view, and tiers, exclude and on_unknown still apply; selection lists only them. An id not indexed for the place, or not in the tiers asked, raises ValueError before anything is downloaded, as does an empty list.

• credentials (mapping, optional) – Credentials for feeds that need an account with their provider, as {provider_id: {field: value}}. They win, field by field, over the environment and the credentials file (transitio.credentials); a provider the index does not list, a field it does not issue or a value that is not a non-empty string raises ValueError before anything is downloaded. A protected feed (IndexedFeed.access == "key") is not downloaded from its producer’s URLs when the index has no access details for it ("protected feed: the index has no access details for it"), transitio cannot send its credentials ("protected feed: its access method is not supported", a cookie included), its URL is not https ("protected feed: its URL is not https"), none or only some of the provider’s fields are set ("protected feed: credentials missing for <provider>", "... credentials incomplete for <provider>") or a value has characters its method cannot carry ("protected feed: credential <field> has characters its method cannot carry"); each reason ends with "; " and the feed’s access_instructions(). Such a feed is read without credentials from the Mobility Database’s hosted copy when its catalogue record names one (with an API token, from its dataset versions first, as an open feed); the reason then goes into the note of a delivered feed, after "from the Mobility Database hosted copy" when the copy was read, and of a failed download. A feed without a copy is skipped with the reason. Otherwise the feed is probed (for expired) and downloaded from its access URL, then, if that fails, from the hosted copy without the credentials. They are sent only to the access URL’s scheme, host and port: a redirect elsewhere carries none, and a redirect to http or holding a credential fails that download. No reason, note, path, sidecar or report holds a credential, beyond what a server writes into the feed itself, which is kept as served.

• modes (str or list of str, optional) – Keep only feeds serving at least one of tram, subway, rail, bus, ferry — decided from the delivered (post-crop) feed’s routes.txt, since the catalog carries no mode metadata. Unknown mode names raise ValueError.

• duplicate_trips ({"drop", "exact", "keep"}, default "drop") – Whether to leave out of the delivered feeds the trips that repeat, or nearly repeat ("drop"), a trip of an earlier delivered feed, as above: "exact" leaves out exact repeats only, and "keep" compares nothing. Another value raises ValueError.

• expired ({"skip", "keep"}, default "skip") – With "skip", on the index paths, an indexed feed whose index service window misses the day (ends before it, or starts after the study day) is skipped before download when a conditional HEAD to the URL the index crawled, carrying the ETag or Last-Modified it recorded, answers 304 Not Modified: the served archive is still the one the index saw, and the reason ends "; unchanged since indexed". Any other answer, or no recorded validator, downloads the feed and leaves the decision to its computed window, as does a feed downloaded as a catalogued dataset (with an API token). "keep" sends no probe and, without when, keeps a feed whose computed window ended; with when the window and day checks still apply.

• repair (bool, default False) – Repair each feed (gtfstidy contract) after the crop, before use; conservative default leaves feeds untouched.

• crop (bool, default True) – Spatially crop each feed to the area: to its polygon when it has one (a place’s boundary included), otherwise to its bounding box. The crop, also run for a route selection, leaves out a trip naming a route the feed lacks, a stop_times row naming a stop it lacks, a trip such rows leave with fewer than two stop_times, and an exact repeat of a trips.txt row (crop_feed()); a feed whose routes.txt or stops.txt has no id column, or whose trips.txt repeats a trip_id with other values, is skipped.

• osm (bool, default True) – Fetch the OSM extract for the AOI. With place, it is fetched after the feeds, for the place’s parts that hold a stop of a delivered feed (the whole place when none does, nothing was delivered or a delivered feed’s stops.txt cannot be read), each part grown by 1.6 km, cafein’s default snap distance. Parts of that area farther than 1.6 km from every delivered stop may lack OSM data, as the extract need only cover the stops’ surroundings. With aoi it covers the area itself, from the index or the catalogue. The crop keeps each trip that serves the area whole, so a delivered feed’s stops can lie beyond the OSM area and get no footpaths in cafein; stops_outside_osm in selection counts them. With osm=False the OSM stage is skipped and the result’s osm_pbf is None, for callers who want only the GTFS feeds; to_pyrosm then raises and to_cafein builds without a walking network. A failed extract download does not abort the call: the feeds are still delivered, osm_pbf and osm_area are None, a UserWarning says so and osm_note holds "OSM extract not fetched: <error>"; to_cafein then builds without a walking network, as with osm=False. Other errors, such as ExtractNotFoundError when no extract covers the area, still raise.

• directory (str or pathlib.Path, optional) – Where the delivered feeds are copied, each as <name>.zip beside its provenance sidecar <name>.provenance.json: the cropped, route-filtered, repaired or deduplicated feed, or the cached version when none of those ran; a feed cut of repeated trips is written over its copy there. The name is the feed id when it is lowercase ASCII letters, digits, _, ~ and -, starting with a letter or digit, at most 100 characters and not a Windows device name (con, nul, com1 and the like); otherwise it is the id’s ASCII form cut to 80 characters, + and the id’s SHA-256, e.g. f-u2f-prazskaintegrovanadoprava+<sha256> for f-u2f-pražskáintegrovanádoprava. A later call delivering the same feed into the same directory replaces its files, and one that delivers it and then leaves it out for repeating other feeds’ trips removes its archive; a feed skipped before delivery leaves a file an earlier call delivered for it in place. Calls running at the same time need directories of their own. Without it the delivered feeds are the files in the cache, an untransformed one the read-only cached version itself. The OSM extract goes here too. It must lie outside the download cache (ValueError).

• index (Index, str, pathlib.Path or False, optional) – The feed index both index paths select from: an Index or the path of one, by default the installed index (transitio.index.refresh()). False, with aoi only, searches the catalogue instead.

• refresh_token – Passed to the catalog and OSM layers; downloads are cached under cache_dir, by default the platform cache. country_code also keeps only that country’s index places for aoi.

• cache_dir – Passed to the catalog and OSM layers; downloads are cached under cache_dir, by default the platform cache. country_code also keeps only that country’s index places for aoi.

• country_code – Passed to the catalog and OSM layers; downloads are cached under cache_dir, by default the platform cache. country_code also keeps only that country’s index places for aoi.

• use_cache (bool, default True) – Serve feeds from the cache as above. False downloads every feed again and, once a download succeeds, deletes the feed’s other versions; when every attempt fails, the version a cached call would use is delivered with a UserWarning.

• progress (bool, default True) – Progress lines and download bars on stderr; a widget bar in Jupyter when ipywidgets is installed (pip install "transitio[notebook]"). Elsewhere a bar shows on a terminal only, and a line names the download instead. False prints nothing.

• **budgets – The validate_feed keyword arguments. A feed with a table that a budget cuts short cannot be cropped and lands in skipped, the reason naming the file and the budget to raise. A reached max_notices_per_file does not stop the crop (the feed’s report then carries notice_limit_reached), but it does stop repair=True. max_total_bytes also bounds a feed read from inside a larger archive. With when, reference_date is the study day; a different one raises ValueError.

Returns:

osm_pbf, validated feeds (paths), merged reports and repair repairs (fix logs, empty without repair=True) per kept feed, paths ({feed id: path} of the same feeds, in the order of feeds), skipped (feed id, reason) pairs, the selection record, on the index paths selections and contained (above), and places, the index places the feeds were selected for (the place, the area’s parts; empty on the catalogue path). Reports merge the local validation of the delivered feed with the hosted report of the published dataset, so after cropping or repair the hosted side describes the pre-transform original. A report’s summary["droppedRows"] lists the rows the crop left out (the dropped_rows of crop_feed()), None for a feed not cropped. selection has one entry per candidate feed, in candidate order: feed_id, name, decision ("delivered" or "skipped"), reason (why it was skipped), note (about a delivered feed: first "cut to <n> of <m> routes" for one cut to a route selection ("cut to <n> selected routes" when its routes.txt was not read), or "delivered whole: selector out of date" or "delivered whole: selector unavailable" for one whose selector was not trusted, as selections details; then why a contained feed was kept, a similar feed, "from the Mobility Database hosted copy" after a failed download, an agency_timezone not equivalent to the zone of most of its stops, e.g. "agency_timezone America/New_York; stops in Pacific/Honolulu", a placeholder calendar, which a left-out version keeps, rows the crop left out, e.g. "dropped 1860 stop_times.txt rows whose stop_id is not in stops.txt" or "dropped 8 exact duplicate trips.txt rows", the repeated trips left out or kept; several join with "; "), index_window (the index’s [start, end]; None undated or on the catalogue path), feed_window (the computed window of a validated download, delivered or not; None otherwise or when unknown), same_as (earlier deliveries of the same archive) and contained_in (the index’s containers a containment skip names), version_of (for a left-out version, {"feed_id", "route_overlap", "stop_overlap"} against the highest-ranked kept version it pairs with, for an undated feed the highest-ranked dated one starting after its placeholder start when one does), duplicate_trips (the number of trips left out as repeats of an earlier delivered feed’s: 0 for a compared feed that lost none, the first included; None for a feed not compared: with duplicate_trips="keep" or fewer than two feeds delivered, one in another time zone, unreadable or over max_total_bytes, and a feed decided before the comparison), fetched_from (where the download came from: "mdb_dataset" a catalogued dataset, "producer" the feed’s own URL, its download_url or one from the Mobility Database or Transitland Atlas, "mdb_latest" the Mobility Database hosted copy; None when nothing was downloaded), download_errors (the failed download attempts before the one that worked, or all of them when none did, joined with "; "; None when none failed or none was made), cache ("downloaded", "reused", "refreshed" or "fallback", how the feed’s version was obtained; None when none was), a reused version’s fetched_from being its first acquisition’s, stops_outside_osm (the delivered feed’s located stops, the stops.txt rows with usable coordinates other than (0, 0), outside osm_area; None without an extract or when its stops.txt cannot be read) and path (the delivered feed). Windows are ISO dates. selections, with tiers, exclude or on_unknown="exclude", has one entry per feed whose route selector was checked, in the order decided: feed_id, selector_state ("complete", "whole_feed" or "unavailable"), trusted, reason (why it was not trusted: "stale", "unavailable" or "route_absent"; None when trusted), kept and dropped (the feed’s routes the selector kept and removed, sorted; None when the feed was not cut, dropped [] for a whole-feed selector, or when its routes.txt was not read), declared_as (the curator predicate of a complete selector made from one, else None) and selected_by (per matched edge {"tier", "selector_state", "route_ids"}). When place is fetched without tiers and its default view (feeds()) holds none of the place’s feeds, or an area’s places hold none of theirs in their views, view_note names them and the tiers that fetch them, and a UserWarning repeats it, e.g. "default view (region: secondary, tertiary) holds none of the place's 2 feeds: f-a (primary), f-b (primary); tiers=['local'] fetches them". When the OSM extract leaves out parts of the place, or delivered stops lie outside its area, osm_note notes them, e.g. "OSM area: 1 of 47 parts (1783 of 2188 km²); 4970 of 10026 located stops outside it", the stops summed over the delivered feeds, with "(stops.txt of 1 feed not read)" added for feeds not counted; when its download failed, it notes that instead, e.g. "OSM extract not fetched: Could not download any of the 1 extracts that contain the area: https://download.bbbike.org/osm/bbbike/Basel/Basel.osm.pbf (timed out)". FetchResult.selection_table() returns the record as a DataFrame. osm_area is the WGS84 geometry the OSM extract was fetched for (None with osm=False or a failed download).

Return type:

FetchResult