Journal scraper definitions for the ContentMine framework.
This repo is a collection of scraperJSON definitions targeting academic journals. They can be used to extract and download data from URLs of journal articles, such as:
All the scrapers in the collection are automatically tested daily as well as every time any scraper is changed. The tests work by having the expected results for a set of URLs stored, and randomly selecting one of those URLs to re-scrape. If the results match those expected the test passes. If the badge is green and says build|passing
, all the scrapers are OK. If the badge is red and says build|failing
, one or more of the scrapers has stopped working. You can click on the badge to see the test report, to see which scrapers are failing and how.
How well the scrapers are covered by the tests is also checked. Coverage should be 100% - this means every element of every scraper is checked at least once in the testing. If coverage is below 100%, you can see exactly which parts of which scrapers are not covered by clicking the coverage
badge below.
Scrapers are defined in JSON, using a schema called scraperJSON which is currently evolving. The current schema is described at the scraperJSON repo.
If your favourite publisher or journal is not covered by a scraper in our collection, we'd love you to submit a new scraper.
We ask that all contributions follow some simple rules that help us maintain a high-quality collection.
Currently these definitions can be used with the quickscrape tool.
All scrapers are released under the Creative Commons 0 (CC0) license.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.