GitHub – oldmoe/litestack

All your data infrastructure is in gem!

Litestack is a Ruby gem that provides an all-in-one solution for web application data infrastructure for Ruby and Ruby on Rails applications. It leverages the power and embeddability of SQLite to provide a mature SQL database, fast cache, powerful job queue, reliable message agent, full-text search engine and metrics platform in one package.

Litestack delivers superior performance, efficiency, ease of use and cost savings compared to traditional approaches that require separate servers and databases. Its embedded database and cache reduce memory and CPU usage, while its simple interface simplifies the development process. Overall, Litestack sets a new standard for web application development and is an excellent choice for anyone who needs speed, efficiency, and simplicity.

You can read more about why litestack can be a good choice for your next web application here,You may also be interested in litestack benchmark.

With litestack, you only need to add a new gem to your application, which will replace many other gems and services. For example, a typical Rails application using litestack will no longer require the following services:

• Database server (e.g. PostgreSQL, MySQL)
• Cache server (such as Redis, Memcached)
• Job processor (e.g. Sidekiq, Goodjob)
• Pubsub server (e.g. Redis, PostgreSQL)
• Full-text search server (e.g. Elasticsearch, Meilisearch)

To make it more efficient, litestack will detect the presence of Fiber-based IO frameworks such as Async (for example, when you use Falcon web server) or Polyphony. It then switches its caching and queuing background workers to fibers (using the semantics of the existing framework). This is done transparently and often results in lower CPU and memory utilization.

Add to litestack The gem line is added to the application’s Gemfile:

To configure a Rails application to run a complete litestack, run:

$ rails generate litestack:install

litestack currently offers six main components

• Wright Database
• Lite cache
• Streamline work
• wired
• Refine search
• literal

litedb is a wrapper for SQLite3 that provides better default configurations, tuned for concurrency and performance. Out of the box, litedb works seamlessly across multiple processes without database lock errors. litedb can be used in a variety of ways, including:

litedb can be used exactly like the SQLite3 gem because litedb inherits from SQLite3

require 'litestack'
db = Litedb.new(path_to_db)
db.execute("create table users(id integer primary key, name text)")
db.execute("insert into users(name) values (?)", "Hamada")
db.query("select count(*) from users") # => [[1]]

litedb provides tight Rails/ActiveRecord integration, which can be configured as follows

In database.yml

adapter: litedb
# normal sqlite3 configuration follows

litedb provides integration with the Sequel database toolkit, which can be configured as follows

DB = Sequel.connect("litedb://path_to_db_file")

litecache is a high-speed, low-overhead caching library that uses SQLite as the backend. litecache can be seamlessly accessed from multiple programs on the same machine. It also features key expiration, LRU-based eviction, and integer value increment/decrement.

require 'litestack'
cache = Litecache.new(path: "path_to_file")
cache.set("key", "value")
cache.get("key") #=> "value"

In your desired environment file (e.g. production.rb)

config.cache_store = :litecache, {path: './path/to/your/cache/file'}

This provides transparent integration using the Rails caching interface

litecache will spawn a background thread for cleanup. If it is detected that the current environment has fiber::scheduler or polyphony Once loaded, it spawns a fiber, saving memory and CPU cycles.

For more information about Litejob, see Streamlining work guide

litejob is a fast and very efficient job queue processor for Ruby applications. It is also built on SQLite, providing transaction guarantees, durability and excellent performance.

require 'litestack'
# define your job class
class MyJob
  include ::Litejob
      
  queue = :default
      
  # must implement perform, with any number of params
  def perform(params)
    # do stuff
  end
end
    
#schedule a job asynchronusly
MyJob.perform_async(params)
    
#schedule a job at a certain time
MyJob.perform_at(time, params)
    
#schedule a job after a certain delay
MyJob.perform_after(delay, params)

In your desired environment file (e.g. production.rb)

config.active_job.queue_adapter = :litejob

You can add more configuration in litejob.yml (or in config/litejob.yml if you want to integrate with Rails)

queues:
    - [default, 1]
    - [urgent, 5]
    - [critical, 10, "spawn"]

The queue needs to contain a name and priority (a number between 1 and 10), and optionally a new flag “spawn”, which means each job will execute in its own concurrency context (thread or fiber)

This is a replacement adapter for replacing the operating cable async and other production adapters (e.g. PostgreSQL, Redis). This adapter is currently only tested in local (inline) mode.

To start and execute Litecable, you need to set the Cable.yaml file in the config/ directory.

cable.yaml

development:
  adapter: litecable

test:
  adapter: test

staging:
  adapter: litecable

production:
  adapter: litecable

Litesearch adds a new full-text search feature to Litedb, you can use it in standalone mode as follows:

require 'litestack/litedb'
db = Litedb.new(":memory:")
# create the index
idx = db.search_index('index_name') do |schema|
    schema.fields [:sender, :receiver, :body]
    schema.field :subject, weight: 10
    schema.tokenizer :trigram
end
# add documents
idx.add({sender: 'Kamal', receiver: 'Laila', subject: 'Are the girls awake?', body: 'I got them the new phones they asked for, are they awake?'})
# search the index, all fields
idx.search('kamal')
# search the index, specific field, partial workd (trigram)
idx.search('subject: awa') 

Litesearch is tightly integrated with ActiveRecord and Sequel. The following is an integration example

class Author < ActiveRecord::Base
    has_many :books
end

class Book < ActiveRecord::Base
    belongs_to :author

    include Litesearch::Model

    litesearch do |schema|
        schema.fields [:title, :description]
        schema.field :author, target: 'authors.name'
        schema.tokenizer :porter
    end
end
# insert records
Author.create(name: 'Adam A. Writer') 
Book.create(title: 'The biggest stunt', author_id: 1, description: 'a description') 
# search the index, the search method integrates with AR's query interface
Book.search('author: writer').limit(1).all

class Author < Sequel::Model
    one_to_many :books
end

class Book < Sequel::Model
    many_to_one :author

    include Litesearch::Model
    litesearch do |schema|
        schema.fields [:title, :description]
        schema.field :author, target: 'authors.name'
        schema.tokenizer :porter
    end
end
# insert records
Author.create(name: 'Adam A. Writer') 
Book.create(title: 'The biggest stunt', author_id: 1, description: 'a description') 
# search the index, the search method integrates with Sequel's query interface
Book.search('author: writer').limit(1).all

Litestack comes with a module that collects useful metrics for its different components. In each component, you need to add the following to the respective .yml file (or database.yml in the case of Litedb)

    metrics: true # default is false

If you enable metrics, it will start collecting data from each module and store them in a database file called metric.db located in the Litesupport.root folder

Litemetric has an API that can collect any metric that is not a litestack category. These indicators will be located in the database, but currently Liteboard can only display correct data for the Litestack module, and will later include displaying arbitrary indicators for other components.

Liteboard is a simple web server that provides a web interface for collected metrics. It should be globally available for use with the helper type.

It allows you to point to a specific indicator database archive or configuration file and then it will display the data from that indicator database

Indicator view example:

• Database size, number of tables and indexes
• Number of read/write queries
• Read/write query ratio over time
• Read/write query time over time
• slowest query
• Most expensive query (total running time = frequency * cost)

• Cache size, percentage of size limit
• Number of entries
• Read/write over time
• Read hits/misses over time
• Most written items
• Most read items

You are welcome to submit bug reports and pull requests on GitHub: https://github.com/oldmoe/litestack.

This gem is provided as open source under the following terms my license.