CERN OpenData Harvester Actor collects the CERN OpenData catalog, bypassing the 10,000 result limit: it reads public sitemaps, converts links to API calls, and, if necessary, downloads additional records by recid ranges. Normalized data is written to a dataset; raw payloads can optionally be stored in a key-value store. ## Input Define in .actor/input_schema.json. All fields are optional and have defaults: - sitemapIndexUrl (string, default https://opendata.cern.ch/sitemap.xml): root sitemap index. - enableSitemapScan (boolean, default true): enable sitemap crawling. - enableRecidScan (boolean, default true): enable searching by recid ranges. - maxWorkers (integer, default 12): Parallel downloads of API pages from sitemaps. - retries (integer, default 5): HTTP error/429/5xx attempts. - recidMax (integer, default 120000): Upper limit of recid for crawling. - recidStep (integer, default 500): Recid step size for search requests. - pageSize (integer, default 200): Search API page size. - skipIds (array of string): List of IDs/slugs to skip (default includes service pages). - storeRaw (boolean, default false): Whether to store raw API payloads in KV. - rawKeyValuePrefix (string, default raw-cern-opendata/): Key prefix in KV when storeRaw=true. ## Output ### Dataset (default) Each record is a normalized object: json { "id": "12345", "title": "Some dataset", "type": "dataset", "experiment": "CMS", "availability": "open", "file_count": 4, "portal_url": "https://opendata.cern.ch/record/12345", "api_url": "https://opendata.cern.ch/api/records/12345", "created": "2020-01-01T00:00:00", "updated": "2020-05-01T00:00:00", "files": [{ "key": "...", "uri": "...", "size": 123, "availability": "...", "checksum": "...", "version_id": "...", "tags": [...] }], "description": "...", "keywords": [...], "collections": [...], "distribution": {...}, "pids": {...}, "publisher": "...", "language": "...", "run_period": "...", "source": "sitemap|recid-range", "bucket_url": "https://opendata.cern.ch/api/files/<bucket>" } Dataset view overview shows: id, title, type, experiment, availability, file_count, portal_url, api_url, created, updated.