queengugl.blogg.se - Db browser for sqlite portable

Db browser for sqlite portable install#
Db browser for sqlite portable update#
Db browser for sqlite portable portable#

This will eliminate the duplicate attribute “ Topic” as there can be only unique keys in a dictionary: keys=response.css('div.section-content div.pull-left div::text').extract() Then We remove the “ Components” attribute from the keys, then we merge keys and values into a dictionary with the zip method. The output you see is the sign that it worked!Īs seen in the Scrapy console part, we will scrap the attributes and their values with the css method, into the variables keys and values. > cursor.execute("CREATE TABLE IF NOT EXISTS fields(field TEXT PRIMARY KEY, desc TEXT)") Now, DROP and CREATE the fields table: > cursor.execute("DROP TABLE IF EXISTS fields") An error message at this point means that the path to the database must be modified. Next, create a connection to the database and a cursor: db = nnect('tuScraper/tuScraper.0.sqlite3') view(response) View response in a browserĪll good! First thing to do is loading the sqlite3 and regexp modules:

Db browser for sqlite portable update#

fetch(req) Fetch a scrapy.Request and update local objects fetch(url) Fetch URL and update local objects (by default, redirects are followed) scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc) You should have this log and the Python prompt: 01:03:43 DEBUG: Telnet console listening on 127.0.0.1:6023 Launch a scrapy shell with a class URL, any class: So, restart a new Python environment DOS command line with the batch you created earlier. Let’s say the database name and path is \tuScraper\ tuScraper.0.sqlite3 You can create this table and leave it empty as we will recreate it and insert rows from the Python console to validate the connection with the database! The history table will be defined the same way:įinally, I decided to add a fields table, to keep the original names of the attributes, with spaces and capitalization:

Db browser for sqlite portable portable#

We can therefore open DB Browser for SQLite Portable and create a sample database, and our first table classes:Įverything will be text but the PK, as we can always modify it later if needed. Last digit is for:įor the column names, let’s use the attributes as they are shown, without spaces. The first 4 digits number (1174) is for the term of the class: 1-YY-semester. The analysis of the URLs also gives us precious information:Įach class has a unique number (here 6891) that we can use as the primary key for the main classes table.

one history table for the variable attributes: Status, Seats Taken, etc.

one main table for the class title, number, description, etc.

It is clear that the only variable attributes are under Class Availability.

Linux: plenty of solutions are availab2le, just dig on gg:.

Db browser for sqlite portable install#

Windows: SQLiteDatabaseBrowserPortable is portable, so install it some place and just launch it.

Working Scrapy Web Spider With Commentsĭatabase initialization Install the DB browser.

Import the Fields into the Fields Table.

Fetch a Page and Send It To the Browser.

Create a batch for initializing the environment.

Installation of Python 2.7 + Scrapy for Windows.