docs/src/mainGenerate Scalding microsite via sbt-microsites (#1623)Sam Ritchie5 months ago
logoAdd scalding logoChris Aniszczyk3 years ago
maple/src/main/java/comMaple fix for HBaseTap, Issue 781Billy Seol1 year ago
projectScalding updates for Scala 2.12 (#1646)Piyush Narang1 month ago
scalding-args/srcUse reflection over Jobs to find serialized classes (#1654)Jackson Davis21 days ago
scalding-avroadd explicit return types to implicit methods and values (#1660)Flavio W. Brasil23 days ago
scalding-benchmarks/src/test/scala/com/twitter/scaldingimport hygiene: remove unused importsTom Dyas2 years ago
scalding-commonsRefactor the cascading code in the typed API into a self contained backend (#1666)P. Oscar Boykin7 days ago
scalding-coreRefactor the cascading planning code out of Grouped (#1670)P. Oscar Boykin4 days ago
scalding-date/srcRemove unnecessary semicolon (#1668)Shunsuke Otani8 days ago
scalding-dbAdd tailrec annotation (#1671)Shunsuke Otani6 days ago
scalding-hadoop-test/srcadd explicit return types to implicit methods and values (#1660)Flavio W. Brasil23 days ago
scalding-hravenScalding updates for Scala 2.12 (#1646)Piyush Narang1 month ago
scalding-jdbc/srcFixesIan O Connell2 years ago
scalding-json/srcMaking side-effecting functions non-nullaryJoe Nievelt11 months ago
scalding-parquetClean up implicit classtagsPiyush Narang7 months ago
scalding-parquet-fixtures/src/test/resourcesAdd partitioned parquet thrift sourcePiyush Narang8 months ago
scalding-parquet-scroogeReplace Manifests with ClassTagsPiyush Narang7 months ago
scalding-parquet-scrooge-fixtures/src/test/resourcesScalding updates for Scala 2.12 (#1646)Piyush Narang1 month ago
scalding-replRefactor the cascading code in the typed API into a self contained backend (#1666)P. Oscar Boykin7 days ago
scalding-serialization/srcRemove unnecessary semicolon (#1668)Shunsuke Otani8 days ago
scalding-thrift-macrosmake ordered serialization stable across compilations (#1664)Flavio W. Brasil10 days ago
scalding-thrift-macros-fixtures/src/test/resourcesBecause, because... fun, the scala compiler has special naming rules it appears when there are leading underscoresIan O Connell1 year ago
scriptsScalding updates for Scala 2.12 (#1646)Piyush Narang1 month ago
tutorialRequest for Scalding release 0.17.0 (#1641)Piyush Narang16 days ago
.gitignoreGenerate Scalding microsite via sbt-microsites (#1623)Sam Ritchie5 months ago
.travis.blacklistAdd scalding-parquet-fixtures to travis blacklistRuban Monu1 year ago
.travis.ymlScalding updates for Scala 2.12 (#1646)Piyush Narang1 month ago
CHANGES.mdRemove two extra changesPiyush Narang1 year ago
COMMITTERS.mdCreate COMMITTERS.mdAlex Levenson8 months ago
CONTRIBUTING.mdGenerate Scalding microsite via sbt-microsites (#1623)Sam Ritchie5 months ago
LICENSEInitial ImportOscar Boykin5 years ago
NOTICEInitial ImportOscar Boykin5 years ago
README.mdRequest for Scalding release 0.17.0 (#1641)Piyush Narang16 days ago
build.sbtSkip scala doc in 2.12 for java classes (#1663)Piyush Narang17 days ago
sbtUpdate Scala and sbt version (#1610)Shunsuke Otani6 months ago
version.sbtRequest for Scalding release 0.17.0 (#1641)Piyush Narang16 days ago

Scalding

Build status Coverage Status Latest version Chat

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Scalding Logo

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._
import com.twitter.scalding.source.TypedText

class WordCountJob(args: Args) extends Job(args) {
  TypedPipe.from(TextLine(args("input")))
    .flatMap { line => tokenize(line) }
    .groupBy { word => word } // use each word for a key
    .size // in each group, get the size
    .write(TypedText.tsv[(String, Long)](args("output")))

  // Split a piece of text into individual words.
  def tokenize(text: String): Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
  }
}

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Please feel free to use the beautiful Scalding logo artwork anywhere.

Contact

For user questions or scalding development (internals, extending, release planning): https://groups.google.com/forum/#!forum/scalding-dev (Google search also works as a first step)

In the remote possibility that there exist bugs in this code, please report them to: https://github.com/twitter/scalding/issues

Follow @Scalding on Twitter for updates.

Chat: Gitter

Get Involved + Code of Conduct

Pull requests and bug reports are always welcome!

We use a lightweight form of project governence inspired by the one used by Apache projects. Please see Contributing and Committership for our code of conduct and our pull request review process. The TL;DR is send us a pull request, iterate on the feedback + discussion, and get a +1 from a Committer in order to get your PR accepted.

The current list of active committers (who can +1 a pull request) can be found here: Committers

A list of contributors to the project can be found here: Contributors

Building

There is a script (called sbt) in the root that loads the correct sbt version to build:

  1. ./sbt update (takes 2 minutes or more)
  2. ./sbt test
  3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

We use Coveralls for code coverage results: Coverage Status

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively, "com.twitter" and 0.17.0.

Current published artifacts are

  • scalding-core_2.11, scalding-core_2.12
  • scalding-args_2.11, scalding-args_2.12
  • scalding-date_2.11, scalding-date_2.12
  • scalding-commons_2.11, scalding-commons_2.12
  • scalding-avro_2.11, scalding-avro_2.12
  • scalding-parquet_2.11, scalding-parquet_2.12
  • scalding-repl_2.11, scalding-repl_2.12

The suffix denotes the scala version.

Adopters

  • Ebay
  • Etsy
  • Sharethrough
  • Snowplow Analytics
  • Soundcloud
  • Twitter

To see a full list of users or to add yourself, see the wiki

Authors:

Thanks for assistance and contributions:

A full list of contributors can be found on GitHub.

License

Copyright 2016 Twitter, Inc.

Licensed under the Apache License, Version 2.0

About FluentSend Feedback