Python数据分析: UFO长啥样？

阅读量:次     Authors: 阳哥 PROJECTS
Matplotlib Pandas Numpy Seaborn

本文首发于我的微信公众号（ID：PyDataLab）。

Table of Contents

前言

真心讲，长这么大，还没有见过UFO长啥样，偶然看到美国UFO报告中心有关于UFO时间记录的详细信息，突然想分析下这些记录里都包含了那些有趣的信息，于是有了这次的分析过程。

当然，原始数据包含的记录信息比较多，我只是进了了比较简单的分析，有兴趣的童鞋可以一起来分析，别忘了也给大家分享下您的分析情况哦。

本次分析的主要内容涉及以下几个方面：

  • UFO长啥样？
  • UFO在哪些地方出现的次数较多？
  • UFO在哪些年份出现的次数较多？
  • 热力图同时显示哪些州和哪些年UFO出现次数最多
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline
plt.style.use('ggplot')

1 数据整理与清洗

df =  pd.read_csv('nuforc_events.csv')

print(df.shape) # 查看数据的结构
print(df.head())

(110265, 13)
             Event_Time  Event_Date    Year  Month   Day  Hour  Minute  \
0  2017-04-20T14:15:00Z  2017-04-20  2017.0    4.0  20.0  14.0    15.0   
1  2017-04-20T04:56:00Z  2017-04-20  2017.0    4.0  20.0   4.0    56.0   
2  2017-04-19T23:55:00Z  2017-04-19  2017.0    4.0  19.0  23.0    55.0   
3  2017-04-19T23:50:00Z  2017-04-19  2017.0    4.0  19.0  23.0    50.0   
4  2017-04-19T23:29:00Z  2017-04-19  2017.0    4.0  19.0  23.0    29.0   

         City State     Shape     Duration  \
0     Palmyra    NJ     Other    5 minutes   
1  Bridgeview    IL     Light   20 seconds   
2      Newton    AL  Triangle    5 seconds   
3      Newton    AL  Triangle  5-6 minutes   
4      Denver    CO     Light       1 hour   

                                             Summary  \
0    I observed an aircraft that seemed to look odd.   
1  Bridgeview, IL, blue light.  ((anonymous report))   
2                               Silent triangle UFO.   
3  My friend and I stepped outside hoping to catc...   
4  Moved slow but made quick turns staying and ci...   

                                           Event_URL  
0  http://www.nuforc.org/webreports/133/S133726.html  
1  http://www.nuforc.org/webreports/133/S133720.html  
2  http://www.nuforc.org/webreports/133/S133724.html  
3  http://www.nuforc.org/webreports/133/S133723.html  
4  http://www.nuforc.org/webreports/133/S133721.html
  • 由于存在许多包含NaN的数据信息，在进行分析之前，先用dropna()方法去除包含NaN的行数
df_clean = df.dropna()
print(df_clean.shape) # 查看去除Nan后还有多少行
print(df_clean.head())

(95004, 13)
             Event_Time  Event_Date    Year  Month   Day  Hour  Minute  \
0  2017-04-20T14:15:00Z  2017-04-20  2017.0    4.0  20.0  14.0    15.0   
1  2017-04-20T04:56:00Z  2017-04-20  2017.0    4.0  20.0   4.0    56.0   
2  2017-04-19T23:55:00Z  2017-04-19  2017.0    4.0  19.0  23.0    55.0   
3  2017-04-19T23:50:00Z  2017-04-19  2017.0    4.0  19.0  23.0    50.0   
4  2017-04-19T23:29:00Z  2017-04-19  2017.0    4.0  19.0  23.0    29.0   

         City State     Shape     Duration  \
0     Palmyra    NJ     Other    5 minutes   
1  Bridgeview    IL     Light   20 seconds   
2      Newton    AL  Triangle    5 seconds   
3      Newton    AL  Triangle  5-6 minutes   
4      Denver    CO     Light       1 hour   

                                             Summary  \
0    I observed an aircraft that seemed to look odd.   
1  Bridgeview, IL, blue light.  ((anonymous report))   
2                               Silent triangle UFO.   
3  My friend and I stepped outside hoping to catc...   
4  Moved slow but made quick turns staying and ci...   

                                           Event_URL  
0  http://www.nuforc.org/webreports/133/S133726.html  
1  http://www.nuforc.org/webreports/133/S133720.html  
2  http://www.nuforc.org/webreports/133/S133724.html  
3  http://www.nuforc.org/webreports/133/S133723.html  
4  http://www.nuforc.org/webreports/133/S133721.html
  • 由于1900年以前的数据较少，这里选择1900年以后的数据来进行分析，如下：
df_clean = df_clean[df_clean['Year']>=1900] # 获取1900年以后的数据来进行分析
  • 查看导入的每列数据的数据类型，通过运行结果，可以看到，“Event_Date”列并不是日期类型，因此要将之转换。
  • 可以采用pd.to_datetime()方法来操作
df_clean.dtypes

Event_Time     object
Event_Date     object
Year          float64
Month         float64
Day           float64
Hour          float64
Minute        float64
City           object
State          object
Shape          object
Duration       object
Summary        object
Event_URL      object
dtype: object
  • 用pd.to_datetime()方法来将str格式的日期转换成日期类型
pd.to_datetime(df_clean['Event_Date']) # 1061-12-31年不能显示
# OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1061-12-31 00:00:00
df_clean.dtypes

Event_Time     object
Event_Date     object
Year          float64
Month         float64
Day           float64
Hour          float64
Minute        float64
City           object
State          object
Shape          object
Duration       object
Summary        object
Event_URL      object
dtype: object

2 UFO长啥样？

  • 按UFO出现的形状类型来分析,统计不同类型的UFO出现的次数
s_shape = df_clean.groupby('Shape')['Event_Date'].count()
print(type(s_shape))
s_shape.sort_values(inplace=True)
s_shape

<class 'pandas.core.series.Series'>

Shape
Changed          1
Hexagon          1
Pyramid          1
Flare            1
Round            2
Crescent         2
Delta            7
Cross          287
Cone           383
Egg            842
Teardrop       866
Chevron       1187
Diamond       1405
Cylinder      1495
Rectangle     1620
Flash         1717
Cigar         2313
Changing      2378
Formation     3070
Oval          4332
Disk          5841
Sphere        6482
Other         6658
Unknown       6887
Fireball      7785
Triangle      9358
Circle        9818
Light        20254
Name: Event_Date, dtype: int64

剔除特殊情况

  • 剔除出现次数少于10次的类型
  • 剔除“Unknown”及“Other”类型
s_shape_normal = s_shape[s_shape.values>10]
s_shape_normal

Shape
Cross          287
Cone           383
Egg            842
Teardrop       866
Chevron       1187
Diamond       1405
Cylinder      1495
Rectangle     1620
Flash         1717
Cigar         2313
Changing      2378
Formation     3070
Oval          4332
Disk          5841
Sphere        6482
Other         6658
Unknown       6887
Fireball      7785
Triangle      9358
Circle        9818
Light        20254
Name: Event_Date, dtype: int64

s_shape_normal = s_shape_normal[s_shape_normal.index.isin(['Unknown', 'Other'])==False]
s_shape_normal

Shape
Cross          287
Cone           383
Egg            842
Teardrop       866
Chevron       1187
Diamond       1405
Cylinder      1495
Rectangle     1620
Flash         1717
Cigar         2313
Changing      2378
Formation     3070
Oval          4332
Disk          5841
Sphere        6482
Fireball      7785
Triangle      9358
Circle        9818
Light        20254
Name: Event_Date, dtype: int64

from matplotlib import font_manager as fm
from  matplotlib import cm

labels = s_shape_normal.index
sizes = s_shape_normal.values

explode = (0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.1)  # "explode" ， show the selected slice

fig, axes = plt.subplots(figsize=(10,5),ncols=2) # 设置绘图区域大小
ax1, ax2 = axes.ravel()

colors = cm.rainbow(np.arange(len(sizes))/len(sizes)) # colormaps: Paired, autumn, rainbow, gray,spring,Darks
patches, texts, autotexts = ax1.pie(sizes, labels=labels, autopct='%1.0f%%',explode=explode,
        shadow=False, startangle=150, colors=colors, labeldistance=1.2,pctdistance=1.05, radius=0.95)
# labeldistance: 控制labels显示的位置
# pctdistance: 控制百分比显示的位置
# radius: 控制切片突出的距离

ax1.axis('equal')  

# 重新设置字体大小
proptease = fm.FontProperties()
proptease.set_size('xx-small')
# font size include: ‘xx-small’,x-small’,'small’,'medium’,‘large’,‘x-large’,‘xx-large’ or number, e.g. '12'
plt.setp(autotexts, fontproperties=proptease)
plt.setp(texts, fontproperties=proptease)

ax1.set_title('Shapes', loc='center')

# ax2 只显示图例（legend）
ax2.axis('off')
ax2.legend(patches, labels, loc='center left', fontsize=9)

# plt.tight_layout()
# plt.savefig("pie_shape_ufo.png", bbox_inches='tight')
plt.savefig('ufo_shapes.jpg')
plt.show()

运行结果如下：

ufo_shapes.jpg

3 UFO在美国那些州（state）出现的次数比较多？

按”State”进行分组运算，统计ufo在各个州出现的次数

s_state = df_clean.groupby('State')['Event_Date'].count()
print(type(s_state))
s_state.head()

<class 'pandas.core.series.Series'>

State
AB     438
AK     472
AL     930
AR     791
AZ    3488
Name: Event_Date, dtype: int64

将分析得到的结果进行可视化显示，如下：

fig, ax1 = plt.subplots(figsize=(12,8))

width = 0.5
state = s_state.index
x_pos1 = np.arange(len(state))
y1 = s_state.values
ax1.bar(x_pos1, y1,color='#4F81BD',align='center', width=width, label='Amounts', linewidth=0)
ax1.set_title('Amount of reporting UFO events by State ')
ax1.set_xlim(-1, len(state))
ax1.set_xticks(x_pos1)
ax1.set_xticklabels(state, rotation = -90)
ax1.set_ylabel('Amount')

fig.savefig('ufo_state.jpg')
plt.show()

运行结果如下：

ufo_state.jpg

从上图可看出，ufo在加州（CA）出现的总次数明显比其他地方多，难道是ufo偏爱加州人民？

4 UFO在哪些年份出现的次数较多？

按”Year”进行分组运算，统计ufo在各个年份出现的次数

# df_clean['Year'].astype(int)
s_year = df_clean.groupby(df_clean['Year'].astype(int))['Event_Date'].count()
print(type(s_year))
s_year.head()

<class 'pandas.core.series.Series'>

Year
1905    1
1910    2
1920    1
1925    1
1929    1
Name: Event_Date, dtype: int64

将分析得到的结果进行可视化显示，如下：

fig, ax = plt.subplots(figsize=(12,20))
# fig, ax1 = plt.subplots(figsize=(12,8))
# fig, axes = plt.subplots(nrows=2, figsize=(12,8))
# fig, axes = plt.subplots(ncols=2, figsize=(18,4))

year = s_year.index
y_pos = np.arange(len(year))
x_value = s_year.values
ax.barh(y_pos, x_value,color='#4F81BD',align='center', label='Amounts', linewidth=0)
ax.set_title('Amount of reporting UFO events by Year ')
ax.set_ylim(-0.5, len(year)-0.5)
ax.set_yticks(y_pos)
ax.set_yticklabels(year, rotation = 0, fontsize=6)
ax.set_xlabel('Amount')

plt.savefig('ufo_year.jpg')
plt.show()

运行结果如下：

ufo_year.jpg

从上图可看出，近年来UFO出现的报告次数最多

5 1997年以后的UFO事件分析

  • 通过上述分析可看出，1997年以前，报告发现UFO的事件相对较少，下面将针对1997年以后的情况进行分析
df_97 = df_clean[(df_clean['Year']>=1997)]
df_97['Year'] = df_97['Year'].astype(int)
# df_97.astype({'Year':int})
print(df_97.shape)
print(df_97.head())

(86041, 13)
             Event_Time  Event_Date  Year  Month   Day  Hour  Minute  \
0  2017-04-20T14:15:00Z  2017-04-20  2017    4.0  20.0  14.0    15.0   
1  2017-04-20T04:56:00Z  2017-04-20  2017    4.0  20.0   4.0    56.0   
2  2017-04-19T23:55:00Z  2017-04-19  2017    4.0  19.0  23.0    55.0   
3  2017-04-19T23:50:00Z  2017-04-19  2017    4.0  19.0  23.0    50.0   
4  2017-04-19T23:29:00Z  2017-04-19  2017    4.0  19.0  23.0    29.0   

         City State     Shape     Duration  \
0     Palmyra    NJ     Other    5 minutes   
1  Bridgeview    IL     Light   20 seconds   
2      Newton    AL  Triangle    5 seconds   
3      Newton    AL  Triangle  5-6 minutes   
4      Denver    CO     Light       1 hour   

                                             Summary  \
0    I observed an aircraft that seemed to look odd.   
1  Bridgeview, IL, blue light.  ((anonymous report))   
2                               Silent triangle UFO.   
3  My friend and I stepped outside hoping to catc...   
4  Moved slow but made quick turns staying and ci...   

                                           Event_URL  
0  http://www.nuforc.org/webreports/133/S133726.html  
1  http://www.nuforc.org/webreports/133/S133720.html  
2  http://www.nuforc.org/webreports/133/S133724.html  
3  http://www.nuforc.org/webreports/133/S133723.html  
4  http://www.nuforc.org/webreports/133/S133721.html

将数据按”Year”和”State”进行分组运算，如下：

df_amount_year = df_97.groupby(['Year', 'State'])['Event_Date'].size().reset_index()
df_amount_year.columns = ['Year', 'State', 'Amount']
print(df_amount_year.head())

   Year State  Amount
0  1997    AB       6
1  1997    AK       5
2  1997    AL       8
3  1997    AR      10
4  1997    AZ     127

import seaborn as sns

df_pivot = df_amount_year.pivot_table(index='State', columns='Year', values='Amount')

f, ax = plt.subplots(figsize = (10, 15))
cmap = sns.cubehelix_palette(start = 1, rot = 3, gamma=0.8, as_cmap = True)
sns.heatmap(df_pivot, cmap = cmap, linewidths = 0.05, ax = ax)

ax.set_title('Amounts per State and Year since Year 1997')
ax.set_xlabel('Year')
ax.set_ylabel('State')
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)
f.savefig('ufo_per_year_state.jpg')

ufo_per_year_state.jpg

  • 上图中，颜色越深的地方，表示UFO事件报告的次数越多。

原始数据来源于美国的UFO事件报告中心。

对我的文章感兴趣的朋友，可以关注我的微信公众号「Python数据之道」（ID：PyDataLab），接收我的更新通知。

 « Previous Next »