How to get rid of duplicates in a table

How does one even know they have duplicates ? or to what extent ? You can use this SQL statement right here:

SELECT
title, url, count(*)
FROM
jobtransparncyprod.JobPostings
GROUP BY
title, url
Having
count(title) > 1

view raw
dupes.sql
hosted with ❤ by GitHub

The key here is that we are using the “group by” clause to aggregate a bunch of data. And after that we are using the “having” filter clause, to add the filter “count(title) > 1” which is just saying “title found more then once”

Now that we have made sure that we actually do have duplicates, lets get down to deleting them.

DELETE dupes
FROM jobtransparncyprod.JobPostings dupes, jobtransparncyprod.JobPostings fullTable
WHERE dupes.url = fullTable.url
AND dupes.title = fullTable.title
AND dupes.id > fullTable.id

view raw
deletedupes.sql
hosted with ❤ by GitHub

The key here is that we are using aliases rather than joins or something else. So you can think of “dupes” and “fulltable” as variables. First we set the values of these variables, then we use the where clause to filter the things we are looking for. At this point if we just ran the query, we would just end up deleting our whole table. Therefor we have a final “and” where we specify that we only want to select the id that greater out of the two, rows that were in “dupes” and “fulltable”.

Docker – Cheat Sheet

The basic commands you need, to be productive with docker:

How do I get a list of all running docker containers ?

  • docker ps

How do I just get all the containers ?

  • docker ps -a

How do I remove a container ?

  • docker rm <container id or name>

How do I see all my images ?

  • docker images

How do I remove an image ?

  • docker rmi <name of image here>

How do I get an image on to my local machine ?

  • docker pull <name of image here>

How do I make a container and run it ?

  • docker run <image name>

How do I run & start a interactive shell  with my container ?

  • docker run -it <image name>

How do I map a port from my container to the outside ?

  • docker run -p <outside port>:<port inside docker container> <image name>

How do I get details about an image ?

  • docker inspect <image name>

How do I look at the logs coming out of a container ?

  • docker logs <container name>

How do I start up a container and leave it running, without it consuming a session ?

  • docker run -d <image name>

How do I build my application, and tag it ?

  • docker build -t <user-name>/<app-name> .

 

4 Reasons why you should choose React for your next project !

1) Don’t Touch the DOM – Imperative Vs Declarative

Hey don’t touch the DOM ! React will do that for you. You might be wondering what I mean by that. You see React shields you from manually interacting (Imperative)  with the DOM using Java Script. Instead React uses something called the “Virtual DOM”. The Virtual DOM is a representation of the actual DOM that React uses to figure out what to actually change on the screen. React will always be like “Oh you made that change ? Let me handle the best way that update“. So when you use React you are using a Declarative way of programming.

2) Hair Balls Vs Lego – Component Architecture

Front end Apps these days are pretty complex, for example take Netflix, Facebook, or AirBnB. They all have complex user interactions and require a large number of cross interacting entities. If you tried to build any one of these applications using vanilla Java-script, and then tried to add more and more features to them, you would end up with a large hair ball of code. That would feature large chunks of CSS, JS, and HTML all over the place. And when anyone ever tried to add a new feature, they would have to copy and past code (the horror !). Components to the rescue ! With components you can encapsulated each and every aspect of your app into little manageable chunks, that can be imported into other parts of your application. Making building new features as easy as building with Lego.

3) One Way Data Flow

Every application has “State“. This can be anything from how many times you clicked a button, to your current permissions inside the app. Now in traditional front end apps this state is spread around different chunks of the app. And are not shareable across different areas of the app. For example if you had a news editor app, it would have to know who has permissions to actually publish the content, and give the user some categories to publish it under.

React deals with this problem by enforcing a centralized state. Once the “State” of the app changes, React automatically reacts and makes the necessary DOM changes to reflect the change. With this centralization you minimize the places where potential bugs show up. And you have a better idea where the bugs are in your application. Since data only flows one through your application, from your “State” to the components.

4) Just the UI

React is a library for building UIs, plain and simple. It does not try to be a large massive framework that has every single feature you could want. Therefor you can just add the extra features you want as third party libraries. This way your app becomes highly customized towards your use, rather than you needing to bend a framework to work the way you want it too.

 

Why you should get started on that side project now !

All programmers ( code carpenters, developers, software engineers, or what ever you like to go by) should constantly be making things, on the side, for free.

Why would anyone do something for free, when they could get paid for it ? Well what if I said that it could lead to more money down the line ? Would you do it ? And if you did do it, would you only do it for the money ?

Now if you became something for purely the purposes of making money, then …. well …. you are probably gonna have a pretty unfulfilling life, full of big houses, and fancy cars … maybe. However I think people should do things because it satisfies them. Doing something that satisfies you feeds you “soul” .

As a code carpenter I make things for a living, that help people out. And in general I like making things in any medium. So I frequently go out of my way to make alot of things, that kind of sort of have a purpose. By doing this I get a better understanding of different things, and develop experiences that will help me build things in the future.

 

5 Projects to Improve Your Resume and Learn To Program

One of the best ways to get started with programming or help in finding a job is doing side projects. They help you understand what goes into making large feature rich applications.

This article will primarily focus on end user driven projects, i.e projects that anyone can use. This means all these projects will have some focus on building a UI interface.

Projects:

  1. A website that allows you to get a feed of all your favorite blogs/ YouTube channels.

    • You usually need to go to Medium, Dev.To, Hacker News, or individual blogs. why can’t there be a service where it just gives me everything I subscribe to in one big feed ?
  2. Task Management System.

    • Instead of using Trello, ToDo List, and Jira, why don’t you make your own mixture of the three ? And have it integration with your favorite calendar of choose.
  3. Job Board.

    • Looking for a job these days is pretty spammy. There are a multitude of job boards that have posting after posting, and offer no information other then the post. Most job boards have more features geared towards recruiters, and have non for job seekers. And most of the time you end up have to make a spread sheet to keep all your application organized. Make something that fixes these problems.
  4. Gift Recommendations.

    • Everyone needs some help looking for a gift. Make a site that scraps Amazon for items, then build a ML model to recommend them to users using affiliate links. Use the information from every recommendation to training the model.
  5. Apply Machine Learning to your domain of choice.

    • Are you a Pokemon card nerd ? or a gear head ? Why don’t you make a ML model that can clarify car parts or Pokemon cards , and then deploy it as an API. You might be wonder how you could possibly do that, and its easier then you think. just do the first few course of arguably one the greatest ML course ever:  https://course.fast.ai/index.html

 

Here is a list of things you will end up learning, or have experience in after completing any of the projects:

  1. Application Architecture:

    1. How to structure your code
    2. What Frameworks to use
    3. How data flows through the application
    4. What data structures to use
  2. UI – Design:

    1. How to plan out the interface of the application
    2. How to build for convince
    3. Getting better at CSS
  3. Database Integration:

    1. How to save data
    2. Work with data
    3. Integrate application logic with data logic
  4. Application Security:

    1. How to make user accounts
    2. Stop others from accessing another users content
    3. Save user passwords
    4. Understand basic attack vectors – SQL Injection , XSS, CSRF, etc
  5. Getting Better at your Frame Works of Choice

How I went about choosing a Deep Learning Framework

The following is a excerpt that was made, as part of my final capstone project.

Introduction

The hardware and software section will be primarily exploring the two key parts in the development of neural networks. Currently the two competing software libraries for the development of neural networks are PyTorch and Tensor Flow. And the two competing hardware platforms to train models is between AMD and Nvidia [6]. In this section I will explore the benefits and disadvantages of each.

Deep Learning Software & Hardware Selection

When looking into developing our model I identified the 2 key choices, software selection and hardware selection. I identified framework selection as a key choice since, it would act as the key building block in constructing the model, and effect how fast I could train them. Where as hardware selection was important since it would be the primary limiting factor in how fast I could train the model, and how complex I could make the model.

Software Selection

Due to the exponential expansion of machine learning (ML) research and computing power seen over the last decade. There has also been an explosion of new types of software infrastructure to harness it. This software has come from both academic and commercial sources. The need for this infrastructure arises from the fact that there needs to be a bridge betIen theory and application. When I looked at what Ire the most popular frameworks, I found it was a mix of strictly academic and commercial driven software. The four main frameworks Ire Caffe, Theano, Caffe2 + PyTorch, and Tensor Flow (TF).

When I went about choosing a framework, I considered three different factors, community, language, and performance. Community was one the biggest factors, since I had no real production experience in doing any sort of large scale ML modeling and deployment. The only framework that fulfilled this need was Google’s Tensor Flow. It had been released in 2015 and had been made available to the open source community. Leading to many academic researchers to contribute and influence its development. Which has resulted in many other companies using it in their production deep learning pipelines. The combination of both software developers and scientists using it has led to a lot of community driven development. This has lead to making it easier to use and deploy. A side effect of this large amount of adoption is the generation of detailed documentation. Written by the community, large amount of personal, and company blogs, detailing how they used TF to accomplish their goals. The only real competitor at the time of writing it this is Facebook’s Caffe 2 + PyTorch Libraries which was just open sourced early this year.

The other factor was the language interface it would use. I wanted an easy to use interface, with which to build out the model. When I looked at what was available, I found that all of the popular frameworks were written in C++ and CUDA, but had a easy to use Python based interface. The only framework out of the four mentioned above, that only had C++ based interface was Caffe.

The most important part of framework selection was the performance aspect. Most if not all ML research and production use cases happen on Nvidia GPU hardware. This is due to Nvidia’s development of their CUDA programming framework for use with their GPUs. It makes parallel programming for their GPUs incredibly easy. This parallelization is what lets the complex matrix operations be computed with incredible speed. There were only two frameworks out of the four I mentioned, that used the latest version of CUDA in its code base. Which were TF and Caffe 2 + PyTorch, however Caffe 2 + PyTorch was not as robust as Tensor Flow in supporting the different versions of CUDA.

In the end I choose to go with TF since it had a better community and CUDA support. I did not choose to go with its nearest competitor, since it was not as well documented, and its community was just starting to grow. Whereas TF has been thoroughly documented and has had large deployments outside of Google (such as at places like LinkedIn, Intel, IBM, and UBER). Another major selling point for TF is the fact that, it is free, continually getting new releases, and has become an industry standard tool.

Deep Learning Software Frame Works
Name Caffe Theano Caffe 2 + PyTorch Tensor Flow
Computational Graph Representation No Yes Yes Yes
Release Date 2013 2009 2017 + 2016 2015
Implementation language C++ Python & C C++ C++, JS, Swift
Wrapper languages N/A Python Python, C++ C, C++, Java, GO, Rust, Haskell, C#, Python
Mobile Enabled NO NO YES YES
Corporate Backing UC Berkeley University of Montreal Facebook Google
CUDA enabled NO YES YES YES
Multi GPU Support NO NO YES YES
Exportable Model YES NO YES & NO YES
Library of pretrained models YES NO YES YES
Unique Features Don’t need to code to define Network First to use CUDA and Computational Graph in Memory Uses the original developers of Caffe and Theano frameworks

 

VISDOM – Error function Visualization Tool

 

PoIrs Facebook ML

Tensor Board – Network Visualization and Optimization Tool

 

Developed by Google Deep Brain

 

 

PoIrs Google ML

Under Active Development No No Yes Yes
 

 

NOTE

The reason as to why PyTorch and Caffe 2 are always mentioned together is because they are meant to be used together. PyTorch is much more focused on research and flexibility. Where as Caffe 2 is more focused on production deployment and inference speed. Facebook’s researchers use PyTorch to prototype models, then translate the model into Caffe 2, using their model transfer tool known as ONIX.

Table 1 A summary of all information of note that I collected during my research

Creating ASP.NET Core MVC Integration Tests – Getting Started

Tests should be created to make your life easier down the line. When you may or my not have to add new functionality or rework some previous code. However Unit tests may not always be enough, so we end up using integration testing as well.

Prerequisites :

*  You might run into some issues when trying to add the necessary using statements. Its a known issue (https://github.com/NuGet/Home/issues/4412).

In this post I will take about creating a test to check if a particular form is properly rendering as HTML.

Here is what your test controller should look like.

public class LoanFormShould
{
[Fact]
public async Task RenderApplicationForm()
{
// Creating a web Host Builder to put in your test server
var builder = new WebHostBuilder()
.UseContentRoot(@"Enter your Path here to project")
.UseEnvironment("Development")
.UseStartup<loans.Startup>()
.UseApplicationInsights(); // Only need this if your using that feature
var server = new TestServer(builder);
var client = server.CreateClient(); // creating a client to make a request
var response = await client.GetAsync("/Apply");
response.EnsureSuccessStatusCode();
var responseString = await response.Content.ReadAsStringAsync(); // Getting the string of HTML
Assert.Contains("New Loan Application", responseString); // Using the Assert Contains to see if the has the right title
}
}

You maybe asking at this point what about our database how will we actually be testing if our application performs CRUD operations properly ? Well for that we are going to being the In-memory database provider for Entity Framework. So that we never actually end up touching our actual DBs.

The only change you have to make once you have added the package is checking if you in development or not in your Startup.cs file. As seen below:

public void ConfigureServices(IServiceCollection services)
{
if (CurrentEnvironment.IsDevelopment()) // Here is where we check
{
services.AddDbContext<AppDbContext>(
options => options.UseInMemoryDatabase()); // And we ask it to use the InMemoryDatabase Option
}
else
{
services.AddDbContext<AppDbContext>(
options => options.UseSqlServer(
Configuration.GetConnectionString("DefaultConnection")));
}
services.AddScoped<ICreditCardApplicationRepository,
EntityFrameworkCreditCardApplicationRepository>();
services.AddMvc();
}

view raw
InMemoryDBTesting.cs
hosted with ❤ by GitHub

And that is it. Now your ready to start integration testing in .NET Core MVC 🙂

WTF is Bubble Sort ?

Bubble sort is a type of sorting algorithm. Its not the best for everything but it does have its uses. It gets its name since it makes the largest values “bubble” to the top. Bubble sort does its job by using pair comparisons. So it takes the current value and compares it to it’s neighbor directly to the right.

Below you can see what I mean. We first take 5 and compare it 3, then ask the question “is 5 greater then 3 ? ” Then if it is, 5 switches places with 3 and if it isn’t then they stay in the same place, and move on to the next value.

BubbleSortEx1

Here you can see a non optimized implementation of it:

// Note this is without any optimizations
const bubblesort = (arr) => {
// We have a loop here to slowly shrink how much of the array we cover
// So that we don't constantly keep looping over the whole array
for(let i = arr.length; i > 0; i){
// This loop, covers the length of the array given to us by the
// loop above.
for(let j = 0; j < i 1; i ++){
// Here we do the comparison that asks the question
// "does the value I am currently at greater then its neighbor ?"
// if it is, then we switch the values around, if it isn't we do nothing
if(arr[j] > arr[j+1]){
let tempVal = arr[j];
arr[j] = arr[j+1];
arr[j+1] = tempVal;
}
}
}
}

view raw
BubbleSort.js
hosted with ❤ by GitHub

 

Map vs Reduce vs Filter in JavaScript

So these methods are part of the “functional” aspect of JavaScript. JavaScript is a strange language in a good way. Before we get into this, know that all these methods above try to replace the “for” loop, or any other type of loop you can think of.

Some of you might be saying, “I like my loops thank you very much, it is in every language out there !”. And yes that is very true, however these make writing a whole loop for something trivial a thing of the past. So lets get on with it, its not as hard as you probably things it is.

All these methods only working on arrays. And they never modify the array you apply them to. They just return a new array.

MAP

As in Mapping “X” to “Y”, or transforming X into Y. In the case of JavaScript you are making a map of your values. You basically just give it a function (could be anonymous) and it applies it to each array element, and forms a new array.

// Array you wanna do the operation on
const peopleArray = [ {Name: "Tim", BankBalance: 10},
{Name: "Leo", BankBalance: 20},
{Name: "Sam", BankBalance: 30} ];
// calling the Map function and having it return its value into
// reformattedArray. Where "obj" represents each object in the
// peopleArray
let reformattedArray = peopleArray.map((obj) =>{
let newObj = {};
newObj.Name = "Mr." + obj.Name;
newObj.BankBalance = obj.BankBalance * 1000;
return newObj;
});
console.log(reformattedArray);
// "reformattedArray" is now: [ { Name: 'Mr.Tim', BankBalance: 10000 },
// { Name: 'Mr.Leo', BankBalance: 20000 },
// { Name: 'Mr.Sam', BankBalance: 30000 } ]
// you can also get another parameter that is passed in. Known as the index
// of the object in the old array (peopleArray in this case)
reformattedArray = peopleArray.map((obj, index) =>{
let newObj = {};
newObj.MyIndex = index;
newObj.Name = "Mr." + obj.Name;
newObj.BankBalance = obj.BankBalance * 1000;
return newObj;
});
console.log(reformattedArray);
// "reformattedArray" is now:[ { MyIndex: 0, Name: 'Mr.Tim', BankBalance: 10000 },
// { MyIndex: 1, Name: 'Mr.Leo', BankBalance: 20000 },
// { MyIndex: 2, Name: 'Mr.Sam', BankBalance: 30000 } ]

view raw
MapsInJS.js
hosted with ❤ by GitHub

REDUCE

As in reducing something to its essence, or compressing something down. In the case of JavaScript you are taking all your values in an array, and compressing them into something useful.

// Here we have a list of all the transactions made by Mike
const transActionsForMike = [ {Name: "Bike", Cost: 5000},
{Name: "Apple Music", Cost: 20},
{Name: "Cook Book", Cost: 30},
{Name: "Azure Hosting", Cost: 60},
{Name: "Phone Bill", Cost: 70}];
// And we try to figure out how much Mike Spent
// "reduce" takes in three main arguments, the Accumulator (acc), the
// currentValue (curr) in the form of an anonymous function. And a inital
// value for the accumulator (in our example its 0). The accumulator acts
// like a bucket that you keep adding things to. Or keep modifying every
// interaction of the reduce function.
// Here we are taking each object in the "transActionsForMike" variable. Then
// we are accessing the cost property of each object, then continuously adding it to
// the accumulator.
// then at the very end we are returning the accumulator that gets dumped into the
// MikesSpending variable.
let MikesSpending = transActionsForMike.reduce((acc, curr) =>{
return acc + curr.Cost;
}, 0);
console.log(MikesSpending) // MikesSpending is: 5180
// Note you can make the accumulator into anything you want. That is, you can set the
// inital property of the accumulator to object or array and continously add things
// to it, to have the desired output.
// Other things that you can have "reduce" pass into your function is:
// accumulator, currentValue, currentIndex, array (yes the whole array)

view raw
ReduceInJS.js
hosted with ❤ by GitHub

FILTER

As in finding something based on a certain set of parameters. I think you guys kinda already know what this particular method does.

const transActionsForMike = [ {Name: "Bike", Cost: 5000},
{Name: "Apple Music", Cost: 20},
{Name: "Cook Book", Cost: 30},
{Name: "Azure Hosting", Cost: 60},
{Name: "Phone Bill", Cost: 70}];
// As seen in Reduce and Map, we can add more elements into the
// anonymous functions such as the index and the whole array
// filter gives us each element in the array. We then have to return
// "true" to keep it in, or "false" to get rid of it.
// In the case below we are keeping the element in the new array we are returning
// if it's cost property is less than 100 or more than 20
let MikesSmallTransactions = transActionsForMike.filter((ele) => {
return ele.Cost < 100 && ele.Cost > 20;
});
console.log(MikesSmallTransactions);
// MikesSmallTransactions is now: [ { Name: 'Cook Book', Cost: 30 },
// { Name: 'Azure Hosting', Cost: 60 },
// { Name: 'Phone Bill', Cost: 70 } ]

view raw
FilterInJS.js
hosted with ❤ by GitHub

I hope this helped you better understand Reduce, Map, and Filter 🙂

Questions People have Asked Me – Part 1

Below are some questions I was recently asked, with my answers. 

Please let me know if any of them are wrong, its a learning opportunity for me 🙂

What is the binary sort algorithm and how does it work? 

The binary sort algorithm (BSA) is used to effectively sort data. It works on the principle of continuously cutting the data set in half, until it finds what it is searching for. However, this algorithm only works on data sets that are already sorted. 

First the BSA checks the middle of the data set and compares the value it is searching for. If this value is the value it is searching for then it stops. However, if it is not equal, it checks if the value it found is bigger or smaller then the value it is trying to find. If the value is smaller, the BSA repeats the process on the left, whereas if the value is bigger it repeats the process on the right. This process is repeated multiple times until the value that the BSA is looking for is found. 

What is recursion and how is it used? 

Recursion in programming is when the program starts calling its self, from inside its self. This programming technique is usually used when a single large program can be solved in smaller parts and has a valid base case. Such as computing the Fibonacci Sequence or traversing a binary search tree. 

What is polymorphism and what is its purpose? 

Polymorphism is an aspect of Object-Oriented Programming where a “object” can take on many different forms, if all the forms are its children.  

Explain when you should use interfaces and when you should use abstract base classes. 

Both interfaces and abstract classes are a type of contract, in the class structures of a software application. Interfaces are a form of contract between two different entities, where you want to separate the functions from the implementations. This is done such that you existing application does not need to change much if a certain part of it is changed. This can be seen in the commonly used repository pattern, which is used to separate data access logic from the business logic. Where as abstract classes are a less extreme version of interfaces, where certain methods defined in it can have real implementations. This allows any child class that inherits from the abstract class to get those method implementations. The difference between the two can be further seen in how their child methods are derived. Since interfaces are “implemented”, where as abstract classes are “extended”. 

When should you use static methods and static variables? And when shouldn’t you use them. 

Static methods and variables can be used from a class without having to instantiate it. This is usually used when, you want to group a set of functionality or utility function together. An example of this is the “Math” class in JAVA, which gives the user all the math related function they need. You wouldn’t want to use them when you would be creating your inheritance-based class structures, most of the time. 

 

Write a SQL statement to create a table called “author” with the columns “id”, “name”, “age” (for MySQL or SQL Server). 

CREATE TABLE author ( 

id int NOT NULL AUTO_INCREMENT, 

name varchar(255) NOT NULL, 

age int, 

PRIMARY KEY (id) 

); 

Write a SQL statement to create a table called “book” with the columns “id”, “title”, “genre”, “author_id” (for MySQL or SQL Server). 

CREATE TABLE book ( 

id int NOT NULL AUTO_INCREMENT, 

title varchar(255) NOT NULL, 

genre varchar(255), 

author_id int, 

PRIMARY KEY (id), 

FOREIGN KEY (`author_id`) REFERENCES `author` (`id`) ON DELETE CASCADE); 

 

For the “author” and “book” tables created above, write a SQL statement to tell you the number of books each author has written, but only for authors who have written 2 or more books. The output should not show authors that have written only 1 book. The output should have the author’s name and the number of books they have written. 

SELECT author.name, COUNT(*) AS ‘# books’ FROM author, book WHERE author.id = book.author_id GROUP BY author.name HAVING COUNT(*) > 1; 

 

In databases, what are indexes used for and how to you decided how to use them effectively. 

Indexes in databases are used to speed up data retrieval. However, they come at the additional cost of space, and added complexity to database maintenance. They should only ever be used when the same type, or group of data is constantly being accessed. If the number of reads get even larger, there should also be some sort of caching layer the application queries, such that it doesn’t need to query the SQL database directly. 

 

What is the value of unit testing and what are some of your strategies for writing good unit tests? 

Unit testing is used to test the functionality of the different parts of an application. Its value lies in the fact that they make the programmer, test their code in a systematic way. And feeds into a workflow where tests are run before anything gets committed to the master branch. I think the best way to write a test case, is to write the test before writing the application logic, since it gets you thinking about what edge/special cases to consider. This is also known as the test-driven development approach.