Demo: Efficient file sharing over IPFS using Textile

Synchronization of large media files like images, videos, or audio in mobile apps can be challenging. Textile helps solve that challenge by combining our decentralized table protocol to keep apps in sync, IPFS data schemas to pin and read data from the network efficiently, and our file processing tool to transform, resample, or encrypt files into target schemas on any client. Try the demo below to see how Textile Threads plus the data schemas can help you create beautiful feeds of structured data over IPFS.

  • Why did you build this?

    Large data sync in a mobile app is a real headache, especially over a decentralized network. We wanted to share our solution so that other developers could take a look, try it out, and hopefully build new applications on top of it. Check out the full textile-go framework for details.

  • Why use IPFS?

    There are a number of benefits to using IPFS over traditional HTTP that can be more or less valuable depending on your use-case. Some of main reasons we decided to build with IPFS include:

    - Immutable data makes decentralized data streams resistant to tampering.
    - High-performance P2P transfer means we can reduce server and network costs.
    - Censorship resistance means we can provide new and better services to users.
    - Content addressable data means we can let users own & control their data but still use it in apps.

  • Why should I use this?

    Textile gives you the benefits of adding decentralized systems to your app without having to solve all the challenges of maintaining multiple data schemas, processing workflows, and storage formats across clients... that's why!

  • Does it only work for photos?

    No! Photos are just what we use the framework for, but we think it could be useful for all kinds of data. Video files, Audio files, GeoJSON, and documents are just a few that come to mind. In Textile Photos we even use it for p2p conversations. For fun, take a look at our examples for syncing log files or streaming live weather data below.

  • Does it only work in mobile apps?

    We built it for mobile first, meaning that it works really well via our mobile framework. Our framework makes it easy to embed an IPFS node inside a mobile application and our APIs help standardize how an app uses that node. But the framework isn't limited to only running on mobile devices, it runs on mobile, desktop, servers... everywhere! If you are interested in accessing the framework as a service, get in touch.

  • How can I get started?

    Well, first try the demo below. It uses a server-based instance of Textile's framework, but should help you imagine how that framework could be integrated into an application running anywhere. Next, be sure to checkout our GitHub repo to dig into the framework.

Live Demo

Simulate the sharing of an image in your mobile app by posting one to IPFS using Textile's file processing, IPFS data schemas, and decentralized tables.
Step 1. Upload an Image
DRAG AND DROP AN IMAGE
UPLOADING...

Step 2. Setup

The data schema to the right defines how our image should be transformed and shared over IPFS. When developing with Textile, you can define or reuse any number of custom schemas.

The schema on the right tells Textile to resample your image into three new sizes, extract the EXIF data into a JSON document, create an encrypted version of your image, and store all of the content on IPFS in a single folder.

Continue: Transform
{
    "name": "photo",
    "pin": true,
    "plaintext": false,
    "links": {
        "raw": {
            "use": ":file",
            "pin": false,
            "plaintext": true,
            "mill": "/blob"
        },
        "favicon": {
            "use": "raw",
            "pin": true,
            "plaintext": true,
            "mill": "/image/resize",
            "opts": {
                "quality": "80",
                "width": "36"
            }
        },
        "thumb": {
            "use": "raw",
            "pin": false,
            "plaintext": true,
            "mill": "/image/resize",
            "opts": {
                "quality": "80",
                "width": "100"
            }
        },
        "small": {
            "use": "raw",
            "pin": false,
            "plaintext": true,
            "mill": "/image/resize",
            "opts": {
                "quality": "80",
                "width": "256"
            }
        },
        "exif": {
            "use": "raw",
            "pin": true,
            "plaintext": true,
            "mill": "/image/exif"
        },
        "encrypted": {
            "use": "favicon",
            "pin": true,
            "plaintext": false,
            "mill": "/image/resize",
            "opts": {
                "quality": "80",
                "width": "36"
            }
        }
    }
}
Intermediate result

Step 3. Transform

To sync media files over a decentralized network, it is often critical to generate smaller image sizes for clients with poor/limited bandwidth. Here, the client generating the new content automatically samples the original file into three new dimensions, each of which will be added to IPFS.

Note: The default is to encrypt every file. In most steps here we override this behavior with plaintext: true so that the raw files will be added to IPFS.

Continue: Extract
{
    "favicon": {
        "use": "raw",
        "pin": true,
        "plaintext": true,
        "mill": "/image/resize",
        "opts": {
            "quality": "80",
            "width": "36"
        }
    },
    "thumb": {
        "use": "raw",
        "pin": false,
        "plaintext": true,
        "mill": "/image/resize",
        "opts": {
            "quality": "80",
            "width": "100"
        }
    },
    "small": {
        "use": "raw",
        "pin": false,
        "plaintext": true,
        "mill": "/image/resize",
        "opts": {
            "quality": "80",
            "width": "256"
        }
    }
}
Intermediate result

Step 4. Extract

Most media files either contain information you don't want to post at all, or contain information that will be useful if available to sync without the full media file. This step extracts the EXIF data into a new JSON document to host on IPFS, removing sensitive geo-information etc.

Continue: Encrypt
{
    "exif": {
        "use": "raw",
        "pin": true,
        "plaintext": true,
        "mill": "/image/exif"
    }
}
Intermediate result

Step 5. Encrypt

Here, we show how the schema definition can result in an automatically encrypted blob that will be posted to IPFS. In most production schemas, you'll add encryption to every step that will be pinned to IPFS, but for this demo we wanted to showcase a few human-viewable files on IPFS too.

Continue: Structured Content
{
    "encrypted": {
        "use": "favicon",
        "pin": true,
        "plaintext": false,
        "mill": "/image/resize",
        "opts": {
            "quality": "80",
            "width": "36"
        }
    }
}
Intermediate result

Step 6. Structured Content

The magic of Textile's data schemas is that at the end of each file processing job you'll end up with structured data posted to IPFS. This makes it easy for clients receiving the data to choose what data they need to sync locally, and which parts can be left on the network for later. For video or audio files, you could encode different bit rates per leaf, or for geospatial data you could enable multiple simplification levels, all defined via simple JSON schemas!

The raw output of each step is pinned as a /d leaf and a metadata /f leaf. Each pinned /f contains useful information such as pointers to source data, a checksum, and a record of the task that created it. Click on the tree generated for your image to explore the associated files and metadata. Read more about the structured file schema on GitHub.

Continue: Synced data table

Step 7: Synced data table

Jobs you create (defined in Step 2 Setup) are published over a private shared table, meaning they become encrypted, streaming, data feeds that can be shared across trusted devices (or not shared at all). With Textile you can easily sync encrypted content or simply consume raw unencrypted data feeds over IPFS.

See a few use-cases for real-time feeds below.

How can you use Textile's file loader?

The file loader is built into the textile-go framework. We use it in every installation of Textile Photos to handle image processing on device and organizing IPFS pinned content across devices. We designed it to help standardize and automate file based interactions with IPFS in almost any use-case. Below are a few more.

Publish and subscribe to real-time weather data over IPFS. Synchronize encrypted log data across trusted devices. Integrate (encrypted) Textile data feeds with external tools.

Command-line Example

Textile's file loader works on mobile devices (Objective-C, Java, and React Native) or as a command-line tool that can be run most places. Check out a quick tour of the command-line interface.
Read More or Retweet It!