Screenshot-to-code README

The project's README from github.com/emilwallner/Screenshot-to-code. As one markdown file: README.md.


A detailed tutorial covering the code in this repository: Turning design mockups into code with deep learning.

Plug: 👉 Check out my 60-page guide, No ML Degree, on how to land a machine learning job without a degree.

The neural network is built in three iterations. Starting with a Hello World version, followed by the main neural network layers, and ending by training it to generalize.

The models are based on Tony Beltramelli's pix2code, and inspired by Airbnb's sketching interfaces, and Harvard's im2markup.

Note: only the Bootstrap version can generalize on new design mock-ups. It uses 16 domain-specific tokens which are translated into HTML/CSS. It has a 97% accuracy. The best model uses a GRU instead of an LSTM. This version can be trained on a few GPUs. The raw HTML version has potential to generalize, but is still unproven and requires a significant amount of GPUs to train. The current model is also trained on a homogeneous and small dataset, thus it's hard to tell how well it behaves on more complex layouts.

Dataset: https://github.com/tonybeltramelli/pix2code/tree/master/datasets

A quick overview of the process:

1) Give a design image to the trained neural network

Insert image

2) The neural network converts the image into HTML markup

3) Rendered output

Screenshot

Installation

FloydHub

Run on FloydHub

Click this button to open a Workspace on FloydHub where you will find the same environment and dataset used for the Bootstrap version. You can also find the trained models for testing.

Local

pip install keras tensorflow pillow h5py jupyter
git clone https://github.com/emilwallner/Screenshot-to-code.git
cd Screenshot-to-code/
jupyter notebook

Go do the desired notebook, files that end with '.ipynb'. To run the model, go to the menu then click on Cell > Run all

The final version, the Bootstrap version, is prepared with a small set to test run the model. If you want to try it with all the data, you need to download the data here: https://www.floydhub.com/emilwallner/datasets/imagetocode, and specify the correct dir_name.

Folder structure

  |  |-Bootstrap                           #The Bootstrap version
  |  |  |-compiler                         #A compiler to turn the tokens to HTML/CSS (by pix2code)
  |  |  |-resources											
  |  |  |  |-eval_light                    #10 test images and markup
  |  |-Hello_world                         #The Hello World version
  |  |-HTML                                #The HTML version
  |  |  |-Resources_for_index_file         #CSS,images and scripts to test index.html file
  |  |  |-html                             #HTML files to train it on
  |  |  |-images                           #Screenshots for training
  |-readme_images                          #Images for the readme page

Hello World

HTML

Bootstrap

Model weights

Acknowledgments

Screenshot-to-code (emilwallner/Screenshot-to-code)

Searching Database…

这是什么?一个神经网络,能把设计稿图片转换成静态网站的 HTML 代码——把生成的代码渲染出来,就能还原出原来的设计。
这是一个能接入我工作流程的应用,还是一个学习项目?这是一个学习项目,不是生产工具:Keras 模型放在 Jupyter 笔记本里,还配有一篇教程,讲解代码是如何工作的。
我怎么把它跑起来?两种方式:点击 FloydHub 按钮,获得一个带环境、数据集和训练好模型的现成工作区;或者在 Jupyter 中打开本地 .ipynb 笔记本,运行 Cell > Run all。
在本地运行它,我的电脑上需要装什么?只需一套 Jupyter 环境就行——模型都是 Keras 笔记本。Bootstrap 版本自带一个小型测试集;要用完整数据,就得另外下载数据集。
我能把自己画的设计稿喂给它吗?只有 Bootstrap 版本能泛化到新的设计稿,而且它是在一个小而同质的数据集上训练的——README 中提醒说,很难说它在更复杂的布局上表现如何。
这种方法是怎么来的?这些模型基于 Tony Beltramelli 的 pix2code,并受到 Airbnb 的 sketching interfaces 和哈佛的 im2markup 的启发。IBM 通过其 PowerAI 平台提供了算力支持。
哪种模型架构效果最好?用 GRU 代替 LSTM——表现最好的模型使用 GRU,仓库中附带的预训练 Bootstrap 权重也是基于 GRU 的。
如果我想看懂代码,有详细的讲解吗?有——作者写了一篇详细教程《Turning design mockups into code with deep learning》,讲解了本仓库中的代码。
Size
50.4 MB
Stars
16.5K
Updated
CommitAug 16
Age
9y
Commits · 90 Days