LEN

Awk 指定分割符 去重
awk -F ',' '!a[$1]++{print}' 举例 : 单条日志格式如下2019/04/17 00...
扫描右侧二维码阅读全文
17
2019/04

Awk 指定分割符 去重

 awk -F ',' '!a[$1]++{print}' 

举例 : 单条日志格式如下

2019/04/17 00:02:52  eth_usdt  11666  4   策略 1 手续费率 3 盘口差 0.1332207361 ask 163.7691152361 bid 163.6358945000 盘口差比例  0.81‰ 对标交易所 ["binance","okex"] ms 1555430572571 原始数据 {"binance":{"asks":[["163.6900000000"]],"bids":[["163.6400000000"]]},"okex":{"asks":[["163.7499984728"]],"bids":[["163.7300000000"]]}}

 [dev@dev root]$ head -n 1000 benchMarkEnd_20190417.log |grep eth_usdt |awk '{print $2","$13","$15","$23}' |awk -F '[{"]' '{print $1 $8","$12","$19","$23}' |awk -F ',' '!a[$1]++{print}' 

    00:00:00,163.2489600000,163.1160505000,163.1900000000,163.1400000000,163.2100000000,163.1900000000
    00:00:01,163.2439585000,163.1160505000,163.1800000000,163.1400000000,163.2100000000,163.1900000000
    00:00:02,163.2539615000,163.1160505000,163.1900000000,163.1400000000,163.2200000000,163.1900000000
    00:00:03,163.2639645000,163.1210490000,163.2000000000,163.1500000000,163.2300000000,163.1900000000
    00:00:04,163.2639645000,163.1210490000,163.2000000000,163.1500000000,163.2300000000,163.1900000000
    00:00:06,163.2539615000,163.1013932824,163.1900000000,163.1106767680,163.2200000000,163.1900000000

需求是 同一秒只取一条记录即可 , 所以 使用 awk -F ',' '!a[$1]++{print}' 根据 $1 时间去去重

最后修改:2019 年 04 月 17 日 06 : 52 PM
如果觉得我的文章对你有用,请随意赞赏

发表评论