Zhixian's Tech Blog

2017-07-09

How to deploy files to Windows using SFTP via Gitlab pipelines

Summary

This blog post describes how you would deploy files to a Windows Server via SFTP using Gitlab pipelines using shared runners.

The practical uptake for this is that you can deploy files for your website to be served by Internet Information Services (IIS) server using Gitlab pipelines.

Note: The context of this post is about deploying websites but the steps described can be used for deploying any type of file using Gitlab pipelines.

Contents

  1. Assumptions
  2. What are Gitlab pipelines
  3. How Gitlab pipelines work
  4. Sample .gitlab-ci.yml

Assumptions

  1. You have an working Gitlab account.
  2. You have a working Gitlab repository.
  3. You have a Windows Server
  4. You have a SFTP server running on your Windows Server and you have a working SFTP account for that server.

If you do not have a SFTP server, you can consider SFTP/SCP Server from SolarWinds.
Its not a fantastic product but it would have to do (considering that it is a free product)
The software is available at the following url after registration:
http://www.solarwinds.com/free-tools/free-sftp-server/registration

What are Gitlab pipelines

To put it simply, pipelines is Gitlab’s mechanism to perform tasks specified by you when you check-in files into your Gitlab repository. These tasks are executed by processes (dubbed "runners" in Gitlab terminology).

The runners can be grouped in shared and private (non-shared) runners.

Shared runners are hosted by Gitlab to be used by all users of Gitlab that wishes to use them). They are free to use but are limited to 2000 CI minutes per month unless you upgrade your Gitlab plan.

In comparison, private runners are setup using your own resources. After you setup your private runner, you have to register it to Gitlab in order to have Gitlab to use it.

How Gitlab pipelines work

When you check in files into your Gitlab repository, Gitlab will check for the existence of a file called ".gitlab-cl.yml". This file must be named exactly as typed (it is case-sensitive). The existence of this file tells Gitlab that there are tasks to be done. This file will list out the "jobs" for Gitlab to carry out.

Side note: As can be guessed from the file extension ".yml", this is a YAML (YAML Ain’t Markup Language) file. For details for the syntax of YAML, see http://www.yaml.org/

Sample .gitlab-ci.yml

As mentioned in the summary of this blog post, we want to setup a Gitlab pipeline that deploy to our SFTP server whenever we checked in a file. As such the below is the ".gitlab-ci.yml" file that would allow us to do that.

image: alpine

before_script:
– apk update
– apk add openssh sshpass lftp

deploy_pages:
stage: deploy
script:
– ls -al
– mkdir .public
– cp -r * .public
– echo "pwd" | sshpass -p $SFTP_PASSWORD sftp -o StrictHostKeyChecking=no zhixian@servername.somedomain.com
– lftp -e "mirror -R .public/ /test" -u zhixian,$SFTP_PASSWORD sftp://servername.somedomain.com
artifacts:
paths:
– .public
only:
– master

The following is what what each of lines do:

Line 1: Declare that "jobs" will be executed in a Docker container that use the image "alpine". The "alpine" image used here is one of the lightest Linux container, Alpine Linux. You can use other images as long as that image is in Docker store.

Line 3: The "before_script" section. Declare the actions to be carried before any jobs are executed in this section.

Line 4: Update the Alpine Linux software package manager, "apk". By default, "apk" is empty. So we need to populate it with the software catalog.

Line 5: Install the "openssh", "sshpass" and "lftp" software packages.

Line 7: Our declaration of a job call "deploy_pages"

Line 8: Indicate that this job is only to be execute in the "deploy" stage.

Quick concept of "stage": Basically, a job are executed in different stages in the order of "build", "test", and "deploy". Jobs in the same stage are executed concurrently (assuming there are sufficient runners to execute the jobs).

Line 9: The "script" section. Actions to be carried for the job are specify under here.

Line 10: List files in the docker container entry point. By default, Gitlab will dump a copy of your code repository at the container entry point. I like to see a list of the files. This is otherwise a frivolous step that is not need.

Lines 11 and 12: Make a directory call ".public" (note the period in front of "public") and copy all files at the entry point into this directory.

ZX: This step is for facilitating lftp at step 14. The problem is that Gitlab will dump a copy of the git repository at the entry point as well. But we don’t want to accidentally deploy the git repository, hence the copying of files to a sub-directory.

Line 13: Start a SFTP session to "servername.somedomain.com" using the account name "zhixian" using password stored in secret variable "$SFTP_PASSWORD".
Execute a SFTP command "pwd" and terminate the SFTP session.

ZX: This step seems frivolous, but is essential to the success of this job.
As mentioned, jobs are executed in a Docker container environment.
Hence, if we initiate any form of connection to a new SSH-based environment, system will prompt us to accept the "fingerprint-key" for that new SSH-based environment.
This line creates SFTP connection and accepts "fingerprint-key" for the SSH-based environment without prompts.

ZX: Note the "$SFTP_PASSWORD". This is a secret variable set under your Gitlab repository "Settings" section, under "Pipelines" subsection.

2017-07-09_001326

If you scroll down, you will see a "Secret variables" section like the below. The password to the SFTP account is specified here.

2017-07-09_001418

Line 14: Executes the "lftp" command. Here, we use the "mirror" feature of lftp. This feature makes a replica of the file structure of the source to the destination.

ZX: Note the "sftp://" prefix in front of the server domain name ("servername.somedomain.com"). It is important to include this to establish SFTP connectivity. If this is not specified, lftp will assume normal FTP.

Line 15: Specify the "artifacts" section. Items listed under the "artifacts" section will be available for download after the job is completed.

Line 16: Specify the "paths" section for the artifacts.

Line 17: Specify that ".public" folder is to be treated as a an artifact made available for download.

Line 18: Specify the branch of code that will cause this job would be executed.

Line 19: Specify the this job is to be executed only when someone checked-in to the "master" branch.

That’s basically all that is needed to get Gitlab to send files to your SFTP server.

References

Configuration of your jobs with .gitlab-ci.yml (https://docs.gitlab.com/ee/ci/yaml/)

Advertisements

2016-09-12

Cannot pull images from docker.io

Filed under: docker — Tags: , , , — Zhixian @ 18:14:09 pm

Summary

  1. You are unable to download docker images from the repository.
  2. You received a network timed out error message.
  3. This issue is probably due to your Docker DNS Server setting. Switch it from Automatic to Fixed to resolve issue.

Details

If you just installed docker in Windows (in my case, it is Windows 10 Pro), you may encounter the following error message when trying to pull a docker image from docker.io:

C:\VMs\Docker>docker pull hello-world
Using default tag: latest
Pulling repository docker.io/library/hello-world
Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/hello-world/images. You may want to check your internet connection or if you are behind a proxy.

image

However, when you open up your browser to navigate to the url (https://index.docker.io/v1/repositories/library/hello-world/images) of the image, you found that you have no problems.

image

This maybe due to an issue with the Network settings of Docker.
Specifically, the problem maybe with the DNS Server setting.
The DNS Server is set to Automatic by default and that DNS server may not be able to find the docker image repository.

image

To resolve this issue, simply set the DNS Server setting to “Fixed”.
For the IP address of the DNS Server, you can probably accept the default of “8.8.8.8” (which points Google’s DNS server)
After clicking on the “Fixed” radio button, click on the “Apply” button to apply your changes.
This will cause Docker to restart.

image

After Docker have restarted, you should find that you are able to pull docker images without any issues.

image

2014-09-01

Running Classic ASP on IIS in Windows 7

Filed under: web application development — Tags: , , , — Zhixian @ 00:06:09 am

This blog post describes how to enable IIS7 in WIndows 7 to run classic ASP scripts.

Assumptions

This guide assumes:

  1. You know how to go to Windows 7 Control Panel.
  2. You know how to go to Internet Information Services (IIS) Manager

Steps to enabling ASP on IIS

1. In the Control Panel, click the item labeled “Programs and Features”.

image

2. In the Programs and Features window, click on the option “Turn Windows features on or off”.

image

3. In the Windows Features dialog, scroll down until you reach the item “Internet Information Services”.
Click the plus sign next to item label to display the menu items under “Internet Information Services”.
Repeat the same steps for “World WIde Web Services” under “Internet Information Services”.
Repeat again for “Application Development Features” under “World Wide Web Services”.

In the list of items under “Application Development Features”, is “ASP”.
Checked the box next to “ASP”.
Click on the “OK” button to apply changes.

image

After you clicked on the “’OK“ button, Windows will proceed to install the classic ASP extension into IIS.

image

After installation is complete, Windows will return you back to the Programs and Features window.

image

Verify installation

If your Internet Information Services (IIS) Manager happens to be open, close it.
If you open your IIS manager, you should see a menu item labeled ASP at server node.
image

Clicking on this item will allow you to configure settings for your classic ASP runtime.

2014-08-31

ASP.NET HTTP 503

Filed under: web application development — Tags: , , , , , , — Zhixian @ 22:47:08 pm

When working in ASP.NET, your web site that had been working fine might suddenly give you the below screen:

image

You are most likely to be receiving this error message because the application pool had stopped.

image

Your web site should start up again after you start the application pool.

image

Web site after application pool was started.

image

2014-08-09

Undefined function sqlsrv_connect()

One of the issues with using Microsoft Drivers 3.0 for PHP is it only works up to PHP version 5.4 while the latest stable version is 5.5.
Attempting to connect to a Sql Server database using the driver on a PHP 5.5 runtime will result in the following message.

image

Some workarounds would be:

  1. Use ODBC instead of Microsoft Drivers 3.0 for PHP.
  2. Downgrade the your PHP runtime from 5.5 to 5.4
  3. Get the source code for Microsoft Drivers 3.0 for PHP and compiled it for PHP 5.5 runtime.
  4. Get one of the re-compiled Microsoft Drivers 3.0 for PHP for PHP 5.5 that are distributed by others in the Internet.

2013-03-02

MongoDB Nutshell

Filed under: web application development — Tags: , , , , , , — Zhixian @ 11:12:03 am

I decided that my previous notes on MongoDB quickstart is too wordy. Here’s a more succinct version.

Up-&-Running

  1. Get and unzip MongoDB files from http://www.mongodb.org/downloads into MongoDB directory (eg. C:\MongoDB)
  2. Create a Data directory in MongoDB directory (eg. C:\MongoDB\Data\Test)
  3. Start MongoDB server using C:\MongoDB\mongod.exe –-dbpath C:\MongoDB\Data\Test
  4. Start MongoDB client using C:\MongoDB\mongo.exe

Mongo Client Command Basics

help – display a summary of commands available
db – indicates the database that you are using
show dbs – list databases
use demo – switch to use database call demo
book1 = { author:”Lewis Caroll”, title:”Alice in Wonderland” }defines a document call book1 with attributes author and title.
db.books.insert(book1) – insert “book1” into a collection call “books”
db.books.find() – list documents (as cursor) in “books” collection
db.books.find({title:”Alice in Wonderland”}) – list documents (as cursor) with title of “Alice in Wonderland” in “books” collection

db.books.find().limit(3) – limits the numbers results in list results
db.books.findOne() – return a document in “books” collection

Other tricks – variables & loops

for (var i = 1; i <= 20; i++) db.things.insert( { x : 4 , j : i } )
var c = db.things.find()
while ( c.hasNext() ) printjson( c.next() )
printjson( c [ 4 ] )

Side Notes

A MongoDB is datastore with many databases. Each database may contain multiple collections of free-form documents.
MongoDB will create collections and databases implicitly upon their first use (insertion of data), hence no creation statements are needed.
All records (documents) in MongoDB are dynamic, hence no schema needs to be defined.

MongoDB Up and Running

Filed under: web application development — Tags: , , , , — Zhixian @ 10:25:03 am

This is my notes to quickly get started with MongoDB under Windows environment.
Most of the material here can be found from the MongoDB Manual.

This post covers the following:

  1. Materials required
  2. Installation
  3. Running MongoDB  Server
  4. Running MongoDB Client

Materials required – 1 required

  1. MongoDB (required)
    Download from http://www.mongodb.org/downloads
    As of date (2013-03-02), the production version is 2.2.3.
    I’m using the Windows 64-bit (2008+) build since this as said to enhance performance when running MongoDB with 64-bit Windows.

Installation

Installation of MongoDB is straight-forward as it is simply a zip file.
Simply extract the contents of the zip file to a convenient location like C:\MongoDB.
The contents of the zip file should look like the below:

image

All the executables related to MongoDB are found in the bin folder.
image

Before proceeding further, you should create folder(s) to hold your data as well.

I try to keep to this convention on naming directories to store MongoDB data:<MongoDB directory>\Data\<application>

Where:

  • <MongoDB directory> – refers to the directory where I extracted the MongoDB files.
  • <application> – refers to the name of the application that MongoDB is used to store data for.

For this post, I will store my data in C:\MongoDB\Data\Quickstart

Running MongoDB Server

MongoDB server is run by executing mongod.exe executable file. Basic execution of this file takes the following syntax:

mongod.exe –-dbpath <data directory path>

<data directory path> refers to the directory that you will store MongoDB data.
As per installation step, we will be using C:\mongodb\data\quickstart for example.

To run MongoDB server from the command-line, go to MongoDB’s bin directory and execute the following command:

mongod.exe –dbpath C:\mongodb\data\quickstart

image

If the directory that you specify does not exists, you may get the following screen:

image

To resolve the error, simply create the directory before running the command again.

The first time, you run mongod.exe you may get a firewall warning like the below:

image

For security, ensure that only the Private networks, such as my home or work network option is checked before clicking on the Allow access button. This should be fine for general local development and testing purposes.

If MongoDB has started, you should see the following screen:

image

The important thing to note, is the last line.
It tells you that MongoDB has started and is waiting for requests on port 28017.

Running MongoDB Client

MongoDB client is run by executing mongo.exe executable file, in the MongoDB directory.

The first time you run this executable, you will get a introductory text as shown in the the following screen.

image

Subsequent executions, will not display the introductory text.

image

2011-07-13

Git as an alternate way to get MinGW

Filed under: computing — Tags: , , , — Zhixian @ 00:41:07 am

In my last blog post, I mentioned about install MinGW to have a C/C++ development environment.
I mentioned that the fastest way is to run the MinGW automated installer.
After I shutdown my PC, I thought of another way to get MinGW working on the PC.
That is, getting a MinGW via Git.

What is Git?

Git is a distributed version control system (DVCS for short).
As the name may suggest, the goal of such a system is to manage the various versions of your documents.
The concept of version control is not exactly new but prior to the advent of DVCSes, they exists mostly as monolithic entities that stand alone.
To manage the different versions of a file, you must be connected to the server to properly manage the various versions.

Pros & Cons of using Git to install MinGW

  1. PRO: Installation process is even more straightforward (you simply have to specify the installation directory).
    CON: Weaker control over the installation process (you cannot elect to install the Fortran or Objective-C options).
  2. PRO: The installer itself contains all the necessary files (hence removing the need to be connected while doing installation).
    CON: You are installing a fixed set of features.
  3. PRO: Comes pre-installed with a default set of useful tools and software.
    CON: Used up much more disk space (1+GB as compared to 308MB for a full MinGW installation)
  4. PRO: You get a DVCS in the process.
    CON: You may not need one in the first place.

The biggest annoyance I find with trying to use Git as development environment is you do not know what exactly is installed.
For example, I would not know if it has objective-C or Fortran compile (no in both cases).

2011-07-09

Installing MinGW

Filed under: web application development — Tags: , , — Zhixian @ 21:50:07 pm

This is a blog post on installing MinGW.

  • Short introduction to MinGW
  • Getting MinGW
  • Installing MinGW using the automated installer

Short introduction to MinGW

MinGW is a contraction for "Minimalist GNU for Windows.
The goal of MinGW is to provide a minimal development environment on Windows based on GNU utilities that are commonly found in Unix/Linux platforms.
Its important to note the MinGW is an environment provided on top of Windows. It is not a replacement for a Unix systems nor is it POSIX (Portable Operating System Interface for Unix) compliant. For a POSIX compliant development environment on Windows, you would have to look at cygwin (http://www.cygwin.com/).

Getting MinGW

To install MinGW, I recommend you use the automated installer that can be found in Sourceforge.
This is the fastest way to get a working MinGW environment.

Note: This installer is a downloader front-end.
That is to say, based on the options that you selected in during the installation process, the installer will download only those selected components over the Internet. This process assumes you are connected to the Internet while doing the installation.

The other (more tedious) way is to download each of the required runtime and install it from them.
MinGW’s website has something on this process here. So I will not repeat that information here.

Installing MinGW using the automated installer

After you download the automated installer, you have have an executable like the following:

Icon

Double-click on this executable to start the installation process.

 

After the installer startup, the first 2 steps are typical of every Windows installation.
Just click on the Next button to proceed.

mingw-installation-step1 mingw-installation-step2

The 3rd dialog may be a little confusing.
Its asking if you want to get the latest version of MinGW files or if you would rather get the version of MinGW files associated with the installer.
This is a matter of necessity/preference really. I typically select using the pre-package repository catalogues (ie. the version of MinGW files associated with the installer). After clicking on the  desired radio button option, click on the Next button to proceed with the next step.

mingw-installation-step3

This step is the EULA (End User License Agreement), click on the Next button to proceed after you read through it.

mingw-installation-step4 

The next step is to decide the location to install MinGW in. The default is MinGW directory.
You can pick another directory but preferably one without spaces in it (eg. C:\GNU\MinGW)
After you decide, click on the Next button to proceed with the next installation step.

 

 

mingw-installation-step5 

The next dialog ask which folder do you want to put the shortcuts for MinGW in your start menu.
Click on the Next button to proceed with the next installation step.

mingw-installation-step6

You select the components that you want to install for MinGW in this step.
The default installation only install the C Compiler.
In addition, I would also recommend getting the C++ Compiler and MSYS Basic System.
Update (2011-07-09): The MSYS Basic System is really a basic system. It does not even come with a text editior.
As such, I am correcting my recommendation to include the MinGW Developer Toolkit option.

If you are need a Fortran compiler or Obj-C compiler, feel free to add those options.
Click on the Next button to proceed with the next installation step.

image

After you selected your desired installation options, the next dialog screen will provide a checklist of items for user to check through the items to install.
After you click on the Install button, the install will proceed to download the packages from the Internet.

mingw-installation-step8 mingw-installation-step10

After all the necessary packages are downloaded, you will get the below dialog.
Click on the Finish button to complete the installation process.

mingw-installation-step11

2011-06-25

Installing Windows Live Essentials on Windows XP as a VirtualBox guest OS

Filed under: computing — Tags: , , , , , — Zhixian @ 08:59:06 am

This is about a firewall issue that I encountered while installing Windows Live Essentials on a Windows XP guest OS.

I was setting up another copy of Windows XP on VirtualBox today and along the way, I decide I want to install Windows Live Essentials. So off I go to Microsoft Download Center and download a copy of the Windows Live Essentials.

Note: You cannot get the latest version of Windows Live Essentials which at this time of writing is Windows Live Essentials 2011. For some reason, Microsoft decide to ditch all the users in the Windows XP space in favour of users in Vista and Windows 7 space…Its almost looking as if this will be the last version of Windows Live Essentials that Microsoft will support for XP. Very sad 😥

Everything was fine prior to running the setup file.

Before running WL installer

Here’s the mysterious part. After I execute the installer, my network connection starts to shutdown.

WL running mod

 

After after a while, I get the following screen.

WL Stopped

Umm….I don’t think that’s the correct response. I WAS connected to the Internet before you came along. Clicking on the details button, I get:

WL Stopped in detail

Umm…Thanks for the advice. Its real helpful. I will take that into consideration once I figure out how to run the installer without it killing off my Internet connection at the same time.

So, what went wrong?

The problem lies with the Windows Firewall on the host OS. Recall that this copy of XP was running as a guest OS on VirtualBox. Hence all network traffic would flow through the host OS and its firewall as well.

Now theoretically, Vista should have notify me when the firewall blocked a connection.
However in this particular instance, it did not. 😦

image

 

And in case you are wondering, no, I did not block all incoming connections.

image

Nonetheless, under the Events Viewer, I see:

image

Hmm…Its time to google…which led me to Microsoft’s TechNet site for troubleshooting firewall. An article there led me to think that I need to enable IPSec and Firewall events audit events. You can find out how to do that by going to Enable IPsec and Windows Firewall Audit Events.

But quite honestly, that did not work for me.Vista still did not notify me when the firewall block a outgoing connection. But the message in the event viewer did changed to:

image

And

image

 

So, what’s next?

By this time, I’m starting to get tired of dealing with the firewall issue not to mention I have run out of concise solutions to this problem.

I decide its time for drastic measures.
So I simply turned off Vista’s firewall.
Then went off to my Windows XP guest OS and install Windows Live Essentials.
After the installation completed, I switched Vista’s firewall back on.

And it worked! Live Messenger and Live Writer all that Live Essentials stuff will continue to work.
So apparently, the issue lies with Windows Live Essentials installer.

Older Posts »

Create a free website or blog at WordPress.com.